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I.  INTRODUCTION 
A.         BACKGROUND 

1.  Simulation  and  Instructional  Technology 

From  the  beginning  of  this  century  simulators  have  been  a  necessary  tool  for  safer  and 
more  efficient  flight  training.  The  trainers  in  use  during  this  period  were  mechanical  mock- 
ups  that  simulated  aircraft  control.  With  advances  in  computer  technology  came  advances 
in  instructional  technology.  The  U.S.  Navy  was  experimenting  with  and  studying  instructional 
media  and  simulation  as  early  as  1941  (Ellis,  1986).  Distance  learning,  video 
teleconferencing,  computer-based  simulations,  CD  ROM  courses,  and  the  Internet  are  all 
advances  in  instructional  technologies.  Most  of  the  technological  advances  are  computer 
based  including  the  use  of  multimedia  computer  video  formats  that  integrate  traditional  video 
with  computer  power. 

We  can  do  things  with  technology  that  are  not  possible  without  it.  For  instance,  we 
can  simulate  flight  for  the  novice;  we  can  show  how  truly  volatile  a  chemical  is  without 
exposing  the  student  to  risk;  and  we  can  put  the  trainee  in  simulated  combat  situations  that 
were  not  possible  before.  Advances  in  instructional  technology  have  provided  us  with  a 
means  to  enhance  the  training  environment  without  putting  the  trainee  in  danger  or  resulting 
in  the  loss  of  valuable  equipment.  The  development  of  "safe,  novel  training  techniques,  which 
could  be  achieved  through  simulation  and  mass  training  media"  was  the  charter  behind  the 
establishment  of  the  Navy's  Special  Devices  Center  in  1941  (Dreves,  1971).  Studies  as  early 
as  the  1940s  proved  that  simulators  could  effectively  reduce  actual  flight  training  time  (Ellis, 
1986).    Flight  training  simulators  led  to  developments  in  other  simulators  that  provided 
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effective  and  safe  training  alternatives  including  nuclear  reactor  trainers  and  surface  ship 
trainers. 

2.  Tactical  Patrol  Craft  Trainer  (TPCT)  Background 

The  TPCT  was  developed  to  fill  a  void  in  the  training  pipeline  of  U.S.  Navy  Officers 
who  would  take  command  of  a  Coastal  Patrol  (PC)  class  ship.  High  stress  autonomous 
decision  making  skills  are  not  part  of  the  normal  training  pipeline  or  experience  level  of  these 
officers.  The  need  was  apparent  for  a  system  that  would  reinforce  selected  training  points  and 
enhance  the  decision  making  capability,  under  stress,  of  the  prospective  Commanding  Officer 
(CO).  Commander  Naval  Special  Warfare  Command1  (COMNAVSPECWARCOM)  needed 
a  training  system  that  would  stress  the  student  in  a  realistic  environment,  enhance  advanced 
decision  making  skills,  and  emphasize  training  elements  of  concern  to  the  Coastal  Patrol  Boat 
(PC)  community. 

The  Integrated  Damage  Control  Training  Technology  (IDCTT),  an  interactive 
multimedia  system  developed  for  training  Damage  Control  Assistants  (DCAs)  on  Naval  Ships, 
provided  "levels  of  reality  and  stress  that  have  not  been  experienced  in  damage  control 
training  before"  (Fuller,  1993).  Developers  decided  that  the  IDCTT  could  be  adapted  to 
create  a  system  capable  of  producing  the  desired  training  environment  for  PC  Commanding 
Officers.  The  IDCTT  is  discussed  in  further  detail  in  the  literature  review  section  of  this 
thesis.  From  a  statement  of  requirements  for  further  training  and  the  use  of  current 
technology,  the  TPCT  was  developed. 


!The  PC  class  ship  is  directly  under  the  control  of  COMNAVSPECWARCOM. 


Systems  Integration  &  Research,  Inc.  (SIR)  was  awarded  the  contract  by 
COMNAVSPECWARCOM  to  assemble  the  full  fidelity  computer  hardware  components, 
develop  and  program2  the  initial  TPCT  scenario,  produce  the  analog  video  segments  for 
incorporation  into  the  scenario,  and  program  the  interface  for  the  Integrated  Bridge  System 
(IBS).3  SIR  was  also  tasked  with  installing  the  system,  conducting  initial  system  testing  and 
evaluation,  and  implementing  the  system.  This  thesis  was  conducted  with  the  support  of 
COMNAVSPECWARCOM  during  the  initial  installation  phase. 

B.  AREA  OF  RESEARCH 

The  purpose  of  this  research  is  to  evaluate  the  TPCT  during  its  installation  phase. 
This  research  is  the  beta  testing  of  an  interactive  multimedia  computer  training  system  prior 
to  general  use.  Beta  testing  is  the  pre-implementation  evaluation  of  a  computer  system  to 
determine  if  hardware  or  software  improvements  are  needed  prior  to  release  to  the  customer. 
This  formative  evaluation  analyzes  the  training  system  while  test  subjects  use  it  before  final 
implementation.  This  evaluation  is  an  important  part  of  computer  system  development. 
Goldstein  (1993)  indicates  that  it  is  essential  for  developers  to  determine,  through  formative 
evaluation,  if  improvements  are  needed  before  a  training  system  is  put  in  operation. 

C.  RESEARCH  QUESTIONS 

The  primary  research  objective  is  to  conduct  formative  evaluation  of  the  trainer  and 
make  recommendations  for  implementation.  Specific  research  questions  are: 


2  The  programing  was  subcontracted  to  Tekamah,  Inc. 

3  The  IBS  is  a  navigation  and  control  system  developed  by  Sperry  Marine,  Inc.  that 
is  in  use  on  the  PC  class  ship. 


1 .  What  system  changes  need  to  be  achieved  prior  to  implementation? 

a.  What  changes  need  to  be  made  in  the  Instructor's  Guide? 

b.  What  changes  need  to  be  made  in  the  Student's  Guide? 

c.  What  software  problems  need  to  be  corrected? 

d.  What  hardware  problems  need  to  be  corrected? 

e.  What  additional  instruction  needs  to  be  conducted  prior  to  using  the  trainer? 

2.  What  initial  indications  of  system  effectiveness  are  produced  during  beta  testing? 

3.  What  variables  need  to  be  collected  to  create  a  database  that  will  allow  future  analysis 
of  training  performance? 

4.  During  initial  beta  testing,  what  variables  are  significant  factors  in  the  successful 
completion  of  the  trainer? 

D.  SCOPE  OF  THESIS 

The  primary  thrust  of  this  thesis  is  a  formative  examination  of  the  TPCT  in  beta  testing 
and  recommendations  for  implementation.  The  thesis  identifies  training  system  improvement 
needs,  initial  indications  of  system  effectiveness,  and  initial  factors  of  successful  trainer 
completion.  This  study  is  not  a  comparative  analysis  nor  does  it  address  cost  effectiveness. 
The  TPCT  is  still  in  the  installation  phase  and  the  analysis  will  be  an  integral  element  of  the 
beta  testing.  The  analysis  does  not  measure  the  ultimate  effectiveness  of  the  TPCT. 

E.  ORGANIZATION  OF  THESIS 

Following  the  introduction,  this  thesis  is  organized  into  five  chapters.  First  the 
literature  review  discusses  those  areas  pertinent  to  the  thesis.  Works  are  reviewed  that  detail 
beta  testing,  simulation  (including  the  IDCTT),  training  and  evaluation,  and  those  works  used 
to  evaluate  decision  making.  The  next  chapter  is  a  detailed  TPCT  description.  This  chapter 


familiarizes  the  reader  with  the  basic  TPCT  concept,  the  physical  components  (system 
hardware),  and  the  software  (including  basic  scenario  description).  This  chapter  provides  the 
reader  with  fundamentals  to  understanding  the  research  that  follows. 

The  research  chapters  begin  by  describing  the  methodology.  The  methodology  that 
was  planned  in  the  research  design  phase  is  contrasted  with  the  actual  methodology  used. 
The  shift  in  methodology  was  an  important  factor  in  the  collection  of  data.  Next  is  a 
presentation  of  the  data  and  analyses  followed  by  conclusions  and  recommendations.  The  last 
section  also  includes  current  status  of  the  recommendations4. 


Recommendations  were  submitted  to  COMNAVSPECWARCOM  during  the 
testing  phase. 


H.  LITERATURE  REVIEW 

The  literature  reviewed  pertaining  to  this  research  is  divided  into  three  sections.  The 
first  section  looks  at  literature  in  the  area  of  training  evaluation.  Second,  beta  testing  and 
computer  system  evaluation  literature  is  reviewed.  Third,  is  a  review  of  the  literature  on 
technology  including  simulation  and  its  use  in  training.  This  section  includes  studies  that  look 
at  the  Tactical  Patrol  Craft  Trainer  (TPCT)  predecessor,  Integrated  Damage  Control  Training 
Technology  (EDCTT),  and  the  literature  on  decision  making  and  the  effects  of  stress  is 
reviewed. 
A.       TRAINING  EVALUATION 

Evaluation  is  an  integral  part  of  a  training  system.  It  is  not  a  stand-alone  element.  It 
must  begin  at  inception  and  continue  through  the  life  of  the  system  to  provide  feedback  for 
system  improvement  and  effectiveness.  Goldstein  (1993)  describes  a  systems  approach  to 
training  that  includes  needs  assessment,  development  and  training,  evaluation,  and  validation 
in  a  looped  feedback  system.  This  approach  would  seem  to  indicate  that  evaluation  has  one 
definitive  and  specific  place  in  the  training  system.  But,  further  exploration  of  each  system 
component  makes  it  apparent  that  each  phase  contains  its  own  element  of  evaluation  that 
feeds  back  into  the  system  (Goldstein,  1993). 

The  evaluation  phase  in  this  model  deals  with  the  summative  and  formative  evaluation 
of  a  training  system.  Summative  evaluation  primarily  looks  at  expected  outcomes  of  a 
completed  system  or  whether  the  system  is  more  effective  than  an  alternative  system. 
Summative  evaluation  also  includes  cost-benefit  analysis.  Formative  evaluation  tests  review 


a  training  system  during  its  design  and  implementation  to  ensure  that  basic  concepts  work. 
The  difference  between  summative  and  formative  evaluation,  and  the  importance  of  both  as 
separate  instruments,  is  important  to  the  evaluation  conducted.  As  noted  by  Goldstein, 
"[m]any  research  problems  result  from  one-shot  evaluation  studies  that  attempt  to  combine 
formative  and  summative  evaluations"  (Goldstein,  1993).  To  have  an  effective  system, 
summative  evaluation  must  follow  formative  evaluation  in  the  process  of  training  system 
development  (Goldstein,  1993).  This  thesis  does  not  attempt  summative  evaluation  and  is 
concerned  only  with  the  formative  evaluation  in  the  implementation  phase. 
B.       COMPUTER  SYSTEM  EVALUATION 

Computer  system  evaluation  can  take  on  many  forms  and  can  occur  at  various  points 
in  the  system's  life  cycle.  In  the  beginning,  hardware  tests  may  include  mechanical  stress  and 
electronic  congruity  testing.  The  software  may  undergo  programmer  logic  checks  that 
determine  if  the  code  operates  as  expected.  Toward  this  end,  there  are  software  tools 
designed  to  test  a  computer  system.  For  example,  Rose  (1996)  prepared  an  extensive  article 
about  the  use  of  automated  testing  tools  in  the  quality  assurance  of  computer  systems.  Her 
assertion  is  that  the  use  of  automated  test  tools  is  a  necessary  element  in  the  testing  cycle  to 
reduce  the  time-intensive  process  of  manual  testing.  But  in  the  article  she  also  emphasizes 
that  the  formative  evaluation,  what  she  calls  beta  testing,  involving  the  end  user,  can  not  be 
completely  replaced  with  automated  testing.  In  fact,  she  indicates  that  the  formative 
evaluation  of  the  user  interface  is  essential  to  the  successful  performance  of  the  system. 

A  procedure  for  evaluating  the  user  interface  performance  of  computer-based 
multimedia  systems  for  training  is  described  by  Reeves  and  Harmon  (1994).  The  procedure, 
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"User  Interface  Dimensions,"  is  based  on  elements  that  evaluate  usability  of  the  system.  They 
present  a  thesis  in  which  the  value  of  a  computer-based  training  system  can  be  evaluated 
based  on  whether  "users  have  a  meaningful  and  purposeful  experience"  while  using  the 
system.  The  procedure  evaluates  user  responses  in  areas  "such  as  ease  of  use,  media 
integration,  and  cognitive  load"  (Reeves  &  Harmon,  1994).  They  set  up  ten  dimensions  that 
are  presented  without  being  numerically  grounded.  The  dimensions  are  either  negative, 
positive,  or  somewhere  in  between.  They  do,  however,  recommend  the  eventual  grounding 
of  these  dimensions  on  a  ten-point  rating  scale. 

Reeves  and  Harmon  (1994)  admit  that  the  dimensions  have  not  been  extensively  tested 
or  completely  validated  and  "should  be  subjected  to  rigorous  expert  review."  But,  they  do 
describe  a  very  detailed  use  of  the  procedure  to  evaluate  a  computer-based  multimedia 
educational  program.  The  test  indicated  that  users  generally  had  a  hard  time  operating  the 
system  and  because  of  this,  the  system's  utility  as  a  training  device  was  questioned.  While 
this  procedure  may  tell  us  something  about  the  usability  of  the  system,  it  still  lacks  some 
elements  that  are  needed  to  completely  evaluate  a  training  system.  The  dimensions  are  not 
instruments  of  summative  evaluation  and  do  not  evaluate  system  effectiveness  in  a 
quantitative  manner  or  in  comparison  with  other  methods  of  teaching  the  same  subject. 

This  thesis  is  a  formative  evaluation  and  the  "User  Interface  Dimensions"  is  used  to 
indicate  areas  that  need  improvement.  The  dimensions  will  help  determine  an  initial  measure 
of  effectiveness  by  evaluating  the  usability  of  the  system. 


C.       SIMULATIONS  AND  INSTRUCTIONAL  TECHNOLOGY 

The  U.S.  military's  use  of  simulation  as  a  training  device  is  extensively  chronicled  in 
a  series  of  articles  edited  by  Ellis  (1986).  Included  in  this  series  are  articles  on  needs 
assessment,  training  system  design,  evaluation,  and  validation.  All  the  articles  emphasize 
computer-based  or  simulator-based  training  that,  when  combined,  resemble  Goldstein's 
(1993)  approach  to  training.  Although  most  of  the  evaluation  procedures  are  summative,  the 
need  for  separate  formative  evaluation  prior  to  implementation  is  emphasized.  A  review  of 
training  devices  and  computer-based  training  in  the  military  is  presented  that  provides  a 
history  of  military  uses,  deficiencies  in  training  devices,  effective  uses  for  computer-based 
training,  and  other  issues  relevant  to  the  development  of  these  systems.  The  following 
sections  are  based  on  findings  by  Ellis  (1986)  and  were  used  to  generate  assumptions  for  this 
thesis. 

1.  Ease-of-use 

Ellis  (1986)  found  that  ease  of  use  and  properly  implemented  training  devices  are 
essential  to  the  effectiveness  of  training  systems.  This  finding  agrees  with  the  assumptions 
behind  the  development  of  Reeves  and  Harmon's  "User  Interface  Dimensions"  and  the  use 
of  them  in  this  thesis.  Johnson  (1994)  used  the  dimensions  to  conduct  a  formative  evaluation 
of  the  JDCTT  during  its  implementation.  The  student  responses  to  the  "User  Interface 
Dimensions"  indicated  that  the  JDCTT  had  a  very  appropriate  user-system  interface.  Johnson 
found  that  the  students  were  receptive  to  the  training  and  found  it  easy  to  use.  Baumann,  et 
al.  (1996a)  conducted  a  program  review  of  the  IDCTT  that  validated  Johnson's  (1994) 
findings.  In  the  program  review  the  researchers  also  used  the  "User  Interface  Dimensions" 
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and  determined  that  the  IDCTT  was  perceived  as  easy  to  use.  During  implementation  of  the 
TPCT,  this  research  also  used  the  "User  Interface  Dimensions"  to  evaluate  the  system's 
usability.  The  assumption  for  this  research  was  that  problems  with  usability  would  inhibit  the 
training  system's  effectiveness. 

2.  Touchscreen  Monitors  and  Graphics 

Although  Johnson's  (1994)  IDCTT  results  were  generally  favorable,  there  were  some 
needed  improvements  noted.  The  touchscreen  monitor  was  difficult  for  students  to  operate 
and  was  evaluated  as  a  major  problem  in  the  system.  Ellis  (1986)  found  that  powerful 
computer  graphics  and  touch  screens  are  important  and  desirable  parts  of  a  computer-based 
training  system.  Because  of  Ellis'  assertion  and  Johnson's  finding,  the  touchscreen  monitor 
for  the  TPCT  was  closely  evaluated  during  this  research.  Touch  sensitivity  must  be  set  to  a 
usable  level  and  the  graphics  for  a  touchscreen  monitor  must  be  integrated  in  a  usable  and 
understandable  way.  If  the  buttons  are  too  small,  the  user  will  have  a  hard  time  selecting  the 
appropriate  response  item.  If  the  graphics  are  hard  to  see  or  not  clear,  the  user  will  be  unable 
to  find  the  appropriate  response  item.  So,  not  only  do  the  graphics  provide  important 
"attention  value"  (Wetzel,  et  al.  1994),  but  they  are  also  an  essential  element  in  the 
interactivity  of  the  system. 

Fletcher's  (1990)  research  indicates  high  utility  for  interactive  video  training.  Fletcher 
(1990)  reviewed  28  studies  and  concluded  that  "higher  levels  of  interactivity  were  associated 
with  higher  levels  of  achievement."  Fletcher  (1990)  also  came  to  the  conclusion  that 
interactive  video-based  training  can  be  more  efficient  and  more  effective  than  conventional 
training  methods.    With  increases  in  technology  comes  the  ability  to  increase  the  graphic 
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capability  of  a  training  system  and  this  should  enhance  realism  in  an  interactive  video-based 
training  system.  This  thesis  is  concerned  with  the  trainees'  perceptions  of  system  realism  and 
how  this  affects  the  interactive  capability  of  the  system.  Other  technological  advances 
including  more  computer  power,  voice  recognition,  and  high  levels  of  fidelity  enable 
computer-based  systems  to  increase  levels  of  realism  (Ellis,  1986). 

3.  Fidelity 

Ellis  (1986)  determined  that  fidelity  is  important  to  the  training  system,  but  the  level 
of  fidelity  must  be  consistent  with  the  training  requirements.  Fidelity  is  also  emphasized  as 
a  major  component  in  simulation  training  technology  by  Wetzel,  et  al.  (1994).  But,  they  also 
caution  that  interactive  training  systems  based  on  video  be  developed  with  a  critical  review 
of  the  need  for  fidelity.  Functional  fidelity  (the  amount  a  system  acts  like  the  real  thing)  is 
considered  an  important  element  in  the  effectiveness  of  a  training  system.  But,  the  question 
is  raised  as  to  whether  simulations  must  also  look  like  the  real  thing  (physical  fidelity)  to  be 
efFective.  Studies  cited  in  Wetzel,  et  al.  (1994)  indicate  that  extremely  low  or  extremely  high 
levels  of  fidelity  can  actually  impede  the  learning  process  for  the  beginning  learner5.  But, 
"[a]s  students  become  increasingly  sophisticated,  better  learning  is  achieved  with  increasingly 
higher  fidelity  instruction  that,  these  students  are  better  prepared  to  process"  (Wetzel,  et  al. 
1994).  The  assumption  in  this  thesis  is  that  the  trainee  is  not  a  "beginning  learner"  and  that 
evaluation  of  both  functional  and  physical  fidelity  will  indicate  if  the  system  is  productive. 


5The  definition  of  "beginning  learner"  as  well  as  definitions  of  high  and  low  levels 
of  fidelity  are  not  expanded  on  by  Wetzel,  et  al.  (1994). 
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4.  Trainee  Perceptions 

The  IDCTT  scenario  was  considered  realistic  and  the  high  fidelity  of  the  system  was 
responsible  for  student  perception  of  the  trainer  as  an  effective  training  method  (Johnson, 
1994).  According  to  Ellis  (1986),  adverse  perceptions  of  the  training  system  by  the  trainee 
will  result  in  poor  training  outcomes.  Johnson  found  that  the  students  were  receptive  to  the 
training  and  concluded  that  this  was  a  factor  in  the  system's  effectiveness.  Baumann,  et  al. 
(1996a)  concurred  with  Johnson's  conclusion  and  noted  that  the  training  was  perceived  as 
valuable.  This  thesis  looks  at  pre-training  perceptions  of  simulation  training  and  follows  up 
with  the  trainees'  perceptions  of  the  TPCT  in  various  areas.  If  the  trainees  perceive  the 
TPCT  as  an  effective  training  method,  the  indication  is  that  there  is  less  possibility  of  the 
system  producing  poor  outcomes.  Negative  perceptions  by  the  trainees  led  this  research  to 
look  for  causes  and  possible  corrections.  Of  particular  interest  is  the  trainee's  perception  of 
stress  induced  by  the  trainer. 

5.  Stress  and  Decision  Making 

The  TPCT  was  designed  as  a  high-fidelity  fast-paced  multimedia  system  that  creates 
a  stressful  environment  in  which  decisions  are  made.  Decision  making  under  stress  is  the  "job 
response"  that  the  training  system  is  intended  to  enhance.  Computer-based  training  can  be 
an  effective  means  to  "fine  tune,  integrate,  and  strengthen  job  responses"  (Ellis,  1986).  But, 
"[o]ne  of  the  most  difficult  topics  to  quantify  in  training  is  the  amount  of  stress  in  an 
environment  and  its  effect  on  an  individual's  ability  to  make  decisions"  (Fuller,  1993).  Stress 
induced  by  the  system  is  intended  to  acclimate  the  trainee  to  the  decision  making  process  in 
a  stressful  environment.   The  acclimation  to  stressful  environments  combined  with  expert 
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knowledge  and  confidence  will  reduce  the  perceived  level  of  stress  and  enhance  performance 
(Heslegrave  &  Colvin,  1996).  But,  the  challenge  in  this  research  was  how  to  determine  if  the 
system  was  producing  appropriate  levels  of  stress. 

The  "NASA-TLX"  was  developed  to  assess  the  relative  importance  of  six  factors  in 
determining  how  much  stress  or  pressure  was  experienced  during  a  task  or  exercise  (Hart  & 
Staveland,  1988).  The  "TLX"  is  a  self-report  mechanism  that  measures  perceived  amounts 
of:  1)  Mental  Demand,  2)  Physical  Demand,  3)  Temporal  Demand,  4)  Performance  Demand, 
5)  Effort  Demand,  and  6)  Frustration.  These  factors  were  validated  as  effective  in  identifying 
variations  in  stress  and  diagnosing  sources  of  stress  within  a  task  (Hart  &  Staveland,  1988). 

The  ability  of  the  "TLX"  to  identify  sources  of  stress  within  a  task  or  system  was  an 
important  factor  in  a  review  by  Hill  (1992).  Four  subjective  rating  measures  of  stress  were 
compared  by  FEU,  and  the  "TLX"  was  determined  to  provide  the  best  source  for  determining 
source  of  stress  in  a  task  or  system  (1992).  Because  of  the  validation  of  the  "TLX,"  Hill's 
assertion  of  its  usefulness,  and  use  of  the  "TLX"  by  Johnson  (1994)  and  Baumann  (1996a), 
it  was  used  in  this  research  to  determine  the  stresses  that  are  created  by  the  TPCT  system. 
Evaluating  the  stress  induced  by  the  TPCT  system  will  determine  if  the  system  is  operating 
as  intended.  But,  it  does  not  address  the  question  of  the  effectiveness  of  these  stressors. 
Effectiveness  will  need  to  be  evaluated  in  future  summative  evaluation. 

6.  Effectiveness  and  Efficiency 

Computer-based  training  provides  a  means  to  effectively  train  otherwise  unsafe  or 
economically  impossible  tasks  (Ellis,  1986).  This  research  is  not  looking  at  outcome-based 
effectiveness  or  cost  savings.  But,  the  literature  review  provides  clues  as  to  where  a  system 
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may  need  evaluation  during  implementation  to  enhance  the  possibility  of  effectiveness  or 
prevent  costly  redesign  of  the  system  after  implementation.  Offering  words  of  caution, 
DeBloois  (1988)  indicates  that  development  of  an  effective  interactive  multimedia  training 
system  is  a  complicated  and  time-consuming  project.  But  through  the  review  of  30  studies, 
DeBloois  (1988)  found  a  variety  of  interactive  multimedia  training  systems  were  effective  and 
efficient.  Of  particular  interest  in  this  thesis  are  the  results  of  the  "VISTA  1982"  study  that 
used  videodisc  training  for  leadership  skills  and  the  "IBM  Management  Training  1986"  study 
that  used  videodisc  for  management  training.  In  each  of  these  studies  DeBlooise  (1988) 
reported  that  the  videodisc  training  was  significantly  more  effective  compared  to  more 
traditional  forms  of  training,  including  role  playing.  Because  the  TPCT  is  designed  to 
enhance  leadership  and  management  skills  (i.e.,  decision  making)  the  results  of  these  studies 
indicate  that  the  use  of  videodisc  training  is  a  reasonable  method  to  use. 

Johnson's  (1994)  summative  evaluation  compared  student  performance  in  the  IDCTT 
trainer  with  performance  in  a  "mock-up"  trainer  system  that  was  already  in  place.  Comparing 
the  performances  of  students  in  the  study,  Johnson  concluded  that  the  IDCTT  was  more 
effective  than  the  system  already  in  use.  Baumann,  et  al.  (1996a)  conducted  a  program  review 
of  the  IDCTT  that  validated  Johnson's  (1994)  findings.  In  the  program  review  the  IDCTT 
was  determined  to  be  mentally  demanding  and  to  produce  high  levels  of  stress  and  frustration 
(Baumann,  et  al.  1996a).  These  researchers  also  concluded  that  the  IDCTT  was  more 
effective  than  the  current  "mock-up"  trainer,  that  multiple  scenarios  are  needed,  and  that  the 
system  provided  valuable  training.  Since  the  TPCT  is  based  on  the  IDCTT,  similar  results 
were  expected  and  similar  measurement  instruments  could  be  used. 
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A  cost  benefit  analysis  was  conducted  of  the  IDCTT  system  for  Naval  Sea  System 
Command  (NAVSEA,  1996).  The  draft  report  concluded  that  the  IDCTT  was  cost  effective 
relative  to  current  training  methods,  and  that  there  was  future  utility  for  the  system.  Included 
in  this  analysis  was  an  endorsement  for  future  scenarios  addressing  the  recommendations  of 
Johnson  (1994)  and  Baumann,  et  al.  (1996a).  The  need  for  scenario  variety  in  the  EDCTT 
prompted  a  goal  for  this  research  to  specifically  look  for  recommendations  for  new  and  varied 
scenarios  for  the  TPCT. 

The  Fuller  Decision  Index  (FDI),  a  quantitative  measure  of  proficiency  for  the  EDCTT, 
was  used  to  support  claims  of  effectiveness.  The  FDI  "is  a  decision  making  quality 
measurement  system  that  was  developed  with  the  goal  of  objectively  and  statistically  deriving 
expert  performance  standards  for  the  IDCTT'  (Fuller,  1993).  Although  the  FDI  was  not  used 
in  this  research,  Fuller's  review  of  decision  making  theory  and  stress  provided  background. 
The  EDCTT  is  the  predecessor  to  the  TPCT,  and  a  review  of  the  literature  on  the  EDCTT 
provided  direction  for  this  thesis. 

Johnson's  (1994)  formative  evaluation  of  the  EDCTT  laid  the  groundwork  for  this 
thesis.  Johnson  evaluated  the  EDCTT  with  the  "User  Interface  Dimensions,"  the  "NASA 
Task  Load  Index"  ("TLX"),  and  survey  responses  from  students.  Students  were  more 
responsive  to  the  interactive  multimedia  training  and  it  "promoted  greater  learning,  produced 
significantly  more  stress,  and  stimulated  students"  (Johnson,  1994).  The  student  responses 
to  the  "User  Interface  Dimensions"  indicated  that  the  EDCTT  had  a  very  appropriate  user- 
system  interface,  and  the  responses  to  the  "TLX"  indicated  perceived  high  levels  of  temporal 
demand  on  the  students.   The  fast  pace  of  the  program  caused  minor  problems  that  were 
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remedied  by  students  conducting  trial  runs  on  the  system.  The  students  indicated  that  the 
system  was  limited  and  there  was  a  need  for  more  scenarios  and  different  ship  classes.  The 
preceding  items  are  not  inclusive  of  Johnson's  findings,  but  they  represent  some  of  the  major 
problems  that  were  kept  in  mind  when  developing  the  methodology  for  this  thesis. 

The  literature  reviewed  in  this  section  provided  the  foundation  from  which  this  research 
was  designed.  Before  the  methodology  and  use  of  the  tools  mentioned  in  the  literature  review 
can  be  discussed,  the  following  chapter  presents  a  complete  description  of  the  TPCT  system. 
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HI.  TACTICAL  PATROL  CRAFT  TRAINER  DESCRIPTION 

The  Tactical  Patrol  Craft  Trainer  (TPCT)  is  an  interactive  multi-media  computer-based 
trainer  designed  to  deliver  a  realistic  full  fidelity  and  analog  video  training  scenario.  The 
trainee  sits  (but  could  stand)  in  front  of  two  20"  touch  screen  computer  monitors  that  create 
a  semi-virtual  environment  simulating  the  actions  of  a  PC  commanding  officer.  Unlike  a  total 
virtual  environment,  the  TPCT  trainee  is  subject  to  the  external  environment.  The  physical 
presence  of  the  instructor,  room  temperature,  and  outside  noises  could  potentially  distract  the 
trainee  during  the  scenario.  The  system  currently  has  one  scenario  and  the  intended  trainee 
is  a  prospective  PC  commanding  officer,  normally  a  Navy  lieutenant,  en  route  to  PC 
command.  The  TPCT  instructor  is  not  intended  to  interact  with  the  trainee  during  the 
scenario  but  is  available  if  problems  arise.  This  chapter  introduces  the  reader  to  the  major 
components  of  the  TPCT  system.  The  three  basic  components  are  discussed:  1)  hardware, 
2)  mission  scenario,  and  3)  software. 
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A.       TPCT  HARDWARE  COMPONENTS 

The  hardware  components  of  the  TPCT,  as  illustrated  in  Figure  1,  consist  of  an 
instructor's  TPCT  rack  that  contains  the  major  components  to  run  the  simulation  and  two 
monitors  for  use  by  the  trainee.  The  TPCT  rack  is  a  multi-component  system  that  runs  the 
simulation.  It  contains  four  networked  Central  Processing  Units  (CPUs),  a  laser  disk  player, 
keyboard  and  mouse,  and  a  14-inch  monitor.  Before  any  training  starts,  all  of  the  system 


Figure  1.  TPCT  hardware  components.  Source:  Instructor  Guide, 
components  must  be  initialized  in  a  precise  process  that  must  follow  a  strict  order.  The 
initialization  process  is  detailed  in  the  instructor's  operating  guide  and  discussed  later  in  this 
chapter.    Together,  these  components  control  all  aspects  of  the  simulation  and  allow  the 
instructor  to  aid  and  monitor  the  trainee. 
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1. 


TPCT  Trainer  Monitor 


The  over-the-counter  (OTC)  monitor,  shown  in  Figure  2,  is  used  as  the  communication 
link  between  the  trainee  and  the  scenario.  During  testing,  this  monitor  is  positioned  to  the  left 
of  the  trainee.  This  20-inch  polished-finish6  monitor  uses  touch  screen  technology.  A  touch 
panel  on  the  screen  is  used  for  input  to  the  system  by  the  trainee.  A  keyboard  and  mouse  are 
also  available  but  were  not  used  during  these  test  because  they  are  currently  considered  back- 
ups to  the  touch  screen  technology.     The  trainee  views  video  segments  relevant  to  the 


Figure  2.  TPCT  Trainer  Monitor  view  during  scenario.  Source:  Student  Guide. 


6Polished-finish  refers  to  the  type  of  glass  screen  on  the  monitor.  Polished-finish 
has  a  glossy  look  to  it  while  an  etched  finish  screen  has  a  non-glare  appearance. 
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scenario  in  the  window  on  the  left  portion  this  screen.  All  of  the  video  is  from  the  viewpoint 
of  the  trainee,  as  if  the  trainee  were  looking  at  the  action.  When  there  is  no  relevant  video 
segment,  the  area  is  black.  Mission  and  ship's  status  information  is  available  either  directly 
on  the  screen  or  through  opening  a  window  by  touching  a  screen  button. 

Status  information  includes  such  features  as  depth  of  the  water,  material  condition, 
course,  and  speed.  The  trainee  can  view  most  of  the  information  without  opening  another 
window.  The  status  board  and  reports  sections  contain  information  that  is  not  adjustable  by 
the  trainee.  All  other  areas  and  functions  on  the  screen  can  be  altered  by  the  student  by 
pressing  a  button  activating  an  on-screen  menu  of  choices  related  to  the  desired  action. 
Figure  3  illustrates  the  procedure  for  lighting  off  or  securing  the  ship's  radar.  If  the  trainee 


Figure  3.  TPCT  Trainer  Monitor  enhanced  view.  Source:  Student  Guide. 
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wants  to  secure  the  radar  system,  he/she  would  press  the  radar  button,  press  "Secure"  and 
then  press  "OK"  The  action  is  not  complete  until  OK  or  CANCEL  is  touched  and  no  other 
action  can  be  taken  by  the  trainee  while  this  window  is  open.  Use  of  the  on-screen  menus  is 
equivalent  to  issuing  an  order.  The  ship  will  respond  to  the  request  even  if  it  leads  the  ship 
into  harm's  way.  Decisions  are  made  by  either  ordering  an  action  or  by  ignoring  a  request 
for  action.  At  times,  the  video  segment  may  recommend  a  course  of  action  and  the  trainee 
can  simply  ignore  the  request.  This  is  equivalent  to  a  negative  response.  If  the  trainee  elects 
to  order  the  action,  it  is  done  through  a  series  of  on  screen  buttons. 

2.  Situation  Display  Monitor 

This  is  an  OTC  20-inch  monitor  with  an  etched  finish  and  touch  screen  capability.  The 
Situation  Display  (see  Figure  4)  shows  the  output  from  the  Integrated  Bridge  System  (DBS), 


Figure  4.  IBS  screen  from  Situation  Display.  Source:  Student  Guide. 
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and  it  provides  information  to  the  trainee  regarding  the  ship's  navigation.  The  only  inputs 
from  the  student  to  this  system  are  requests  for  information,  adjusting  views  (zooming  in  or 
out),  or  selecting  plans  entered  previously  by  the  TPCT  system  for  voyage  management.  The 
DBS  uses  information  from  the  TPCT  system  to  indicate  the  ship's  movement  through  the 
water.  The  TPCT's  IBS  display  is  the  same  that  is  found  onboard  the  PC  class  ship.  During 
operation  of  the  scenario,  the  grid  area  contains  an  area  map  representation,  a  visual 
representation  of  the  ship,  and  the  ship's  movement.  This  screen  is  positioned  to  the  right  of 
the  trainee  during  testing. 

3.  TPCT  Rack  14-inch  Monitor 

The  14-inch  instructor's  monitor  in  the  TPCT  rack  allows  the  instructor  to  follow  the 
trainee  through  the  scenario.  The  instructor  can  watch  reproductions  of  either  the  Situation 
Display  Monitor  or  the  Trainer  Monitor  and  can  shift  between  the  two  during  the  training 
session.  The  instructor  sees  what  the  trainee  sees  but  also  sees  information  not  available  to 
the  trainee.  For  example,  the  instructor  can  observe  the  trainee's  progress  into  a  decision 
point  area,  indicated  only  on  this  screen  and  monitor  progress  without  intruding  on  the 
physical  space  of  the  trainee.  The  monitor  is  also  used  during  system  initialization  to  conduct 
system  analysis  and  testing.  It  can  be  used  to  monitor  the  actions  of  any  of  the  four  CPUs. 

4.  Laser  Disk  Player 

The  laser  disk  player,  a  generic  Over  the  Counter  (OTC)  piece  of  equipment,  is  used 
to  feed  the  system  with  analog  video  produced  to  enhance  realism  in  the  scenario.  The 
current  scenario  contains  video  segments  that  appear  only  during  appropriate  times  within  the 
scenario.  There  is  not  a  continuous  video  feed,  and  the  laser  disk  is  accessed  only  according 
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to  scenario  programing  requirements.  The  video  segments  were  professionally  produced  re- 
creations of  events  filmed  onboard  a  PC  class  ship  using  a  combination  of  actors  and  Navy 
members. 

5.  Rack  Mounted  Keyboard  and  Mouse 

A  standard  OTC  personal  computer  keyboard  and  mouse  device  are  provided  for 
instructor  input  into  the  system.  The  mouse  allows  for  standard  click  and  drag  computer 
operations. 

6.  TPCT  Trainer  CPU 

The  trainer  is  an  OTC  Pentium®-based  CPU  running  a  Windows®  NT  operating  system. 
The  TPCT  trainer  scenario  software  is  run  on  this  CPU.  This  unit  also  contains  software  that 
directs  the  actions  of  all  CPUs  and  the  laser  disk  player  in  a  coordinated  fashion  as  required 
by  the  scenario  software. 

7.  Situation  Display  CPU 

This  CPU  controls  the  Situation  Display  Monitor.  It  is  a  Pentium®-based  CPU 
running  an  OS2  operating  system  provided  by  Sperry,  Inc.  The  Situation  Display  Monitor 
is  the  useable  and  visual  output  of  the  IBS.  The  IBS  also  uses  the  next  two  CPUs. 

8.  Simulation- Vision  Technology  CPU 

This  Simulation- Vision  Technology  (Sim-VT)  CPU  interacts  with  all  other  CPUs  and 
the  laser  disk  to  control  the  video  throughout  the  simulation.  This  CPU  also  provides  the  IBS 
navigation  data  to  the  Voyage  Management  System- Vision  Technology  (VMS-VT).  It  is  a 
Pentium®-based  CPU  running  an  OS2  operating  system  provided  by  Sperry,  Inc. 
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9.  Target  Correlator  CPU 

This  CPU  runs  the  Target  Correlator  Monitor  (TCMON)  and  the  Position  Filter 
Module.  This  is  the  remaining  component  of  the  IBS  using  a  Pentium®-based  CPU  running 
an  OS2  operating  system  provided  by  Sperry,  Inc. 
B.       MISSION  SCENARIO 

The  TCPT  scenario  simulates  a  U.S.  Navy  Sea,  Air,  Land  (SEAL)  team  insertion 
mission  using  the  PC  class  ship  as  the  insertion  platform.  The  scenario  begins  with  a  mission 
brief  to  the  trainee  who  is  acting  as  the  Commanding  Officer  of  a  PC.  The  trainee  is  provided 
with  a  computerized  mission  brief  package  that  can  be  read  prior  to  mission  start  and  can  be 
referred  to  during  the  mission.  The  trainee  is  then  briefed  in  a  video  segment  that  simulates 
the  actual  brief  that  might  occur  by  an  operational  commander  prior  to  a  mission.  Both  briefs 
contain  mission  plans,  weather  reports,  enemy  movement  and  capability  reports,  Standard 
Operating  Procedures  (SOPs),  and  other  mission-critical  information.  After  the  mission  brief, 
the  system  pauses  until  the  trainee  indicates  readiness  to  start  by  pushing  the  "Start  Mission" 
button  on  the  TPCT  trainer  monitor. 

The  mission  begins  with  video  simulating  walking  aboard  the  ship  while  the  crew  is 
making  preparations  to  get  underway.  From  this  point  forward,  the  trainee  is  required  to 
make  decisions  based  on  computer-generated  situations  to  complete  the  mission.  The  trainee 
must  make  the  decision  to  get  underway  and  order  it  by  pressing  the  correct  button  on  the 
Trainer  Monitor.  All  of  the  actual  actions  required  by  ship's  company  to  get  underway  are 
then  carried  out  correctly  without  further  monitoring  by  the  trainee.  To  complete  the  mission, 
the  trainee  must  exit  the  friendly  harbor,  transit  to  enemy  waters,  insert  a  SEAL  team  in  an 
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enemy  harbor,  stand  off  while  the  team  conducts  a  mission,  come  back  in  and  pick  up  the 
team,  and  exit  enemy  waters.  The  SEAL  mission  ashore  is  not  a  part  of  the  training  scenario. 
The  trainee  is  expected  to  already  have  the  knowledge  base  required  to  conduct  this  mission. 
But,  this  seemingly  simple  series  of  events  becomes  complicated  rather  quickly. 

The  mission  scenario  is  conducted  under  strict  transit  time  requirements  while  dealing 
with  a  variety  of  problems  and  decisions  along  the  way.  The  problems  and  their  solutions  are 
known  as  "Decision  Points"  or  "Teaching  Points."  The  TPCT  focuses  the  trainee's  attention 
on  an  abundance  of  information  requiring  timely  response  to  the  many  significant  decision 
points  associated  with  a  complex  scenario.  Depending  on  the  nature  of  the  response,  the 
scenario  either  shifts  toward  resolution  or  degradation  that  can  lead  to  prematurely  ending 
the  scenario.  Some  events,  video  segments,  and  sound  bites  are  included  in  the  scenario  to 
create  a  fast-paced  stressful  environment  under  which  the  trainee  must  make  important 
decisions.  The  scenario  ends  as  either  a  mission  success  from  the  correct  treatment  of 
Decision  Points  or  discussing  with  the  instructor  the  appropriate  decisions  to  emphasize  a 
Teaching  Point. 

An  incorrect  response  to  one  of  the  Decision  Points  in  the  hour-long  simulation7  will 
result  in  ending  the  program  either  through  some  direct  catastrophic  situation  or  inability  to 
meet  the  transit  time  requirements.  This  is  not  considered  a  failure,  rather  it  becomes  a 
Teaching  Point  that  can  be  discussed  between  the  instructor  and  trainee.  The  trainee  can  then 


7The  mission  in  this  scenario  would  take  over  6  hours  to  complete.  The  simulation 
is  shortened  through  the  use  of  "warping,"  which  advances  the  trainee  and  ship's  position 
in  the  mission  at  an  accelerated  rate.  During  this  time  no  actions  are  taken  by  the  trainee. 
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restart  the  mission  from  the  beginning  and  again  attempt  mission  completion.  The  scenario 
used  in  this  evaluation  was  not  alterable,  and  there  was  no  way  *  r  the  trainee  to  start  where 
the  last  attempt  ended. 
C.       TPCT  SOFTWARE  COMPONENTS 

Software  in  a  multimedia  system  refers  to  the  programming  code  that  causes  the  system 
to  function.  This  thesis  does  not  evaluate  the  programming  except  in  reference  to  how  well 
it  operates.  The  programming  code,  per  se,  is  not  evaluated.  Flaws  in  the  code  were  noted 
if  a  failure  occurred  during  operation  of  the  system.  Each  CPU  in  the  system  contains  its  own 
set  of  code.  The  three  CPUs  that  operate  the  IBS  were  pre-programmed  by  Sperry,  Inc.  and 
contained  no  unique  programming.  The  programming  for  the  TPCT  Trainer  CPU  was  new 
and  contains  the  innovative  networked  link  to  the  IBS  system.  This  CPU  facilitates  the  multi 
media  interactive  service. 

Integral  to  the  software  is  the  documentation  for  the  system  including  the  operating 
guide,  the  instructor  guide,  and  the  student  guide.  These  documents  were  evaluated  as  part 
of  this  thesis.  They  were  draft  documents  that  needed  testing  and  revision.  The  operating 
guide  served  to  provide  instructions  to  the  instructor  for  operating  the  system,  primarily  for 
starting  the  CPUs  and  securing  them  after  training.  The  instructor  guide  gave  detailed 
scenario  and  training  information  with  procedures  for  conducting  the  training.  The  student 
guide  provided  the  student  with  a  system  overview  and  information  on  how  to  operate  the 
system.  All  of  these  documents  were  draft  copies. 
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IV.  METHODOLOGY 


This  chapter  describes  the  methodology  used  in  this  research.    The  methodology 
evolved  as  a  result  of  pre-testing  the  Tactical  Patrol  Craft  Trainer  (TPCT).  An  explanation 
of  this  evolution  is  provided.    Detailed  in  this  chapter  are  the  subjects,  data  collection 
instruments,  and  procedures  used  to  conduct  the  beta  testing. 
A.       SUBJECTS 

The  proposed  methodology  was  originally  designed  to  compare  the  performance  of 
approximately  10  Subject  Matter  Experts  (SMEs)  with  two  different  groups  of  trainees:  1) 
the  four  current  Patrol  Craft  (PC)  Commanding  Officers  in  San  Diego,  and  2)  about  30  Navy 
Lieutenants8.  The  subject  matter  experts  are  Naval  officers  who  have  had  command  at  sea 
and  thus  could  establish  a  baseline  for  evaluating  the  performance  of  trainees  on  the  system. 
The  assumption  was  that  the  SMEs  would  complete  the  TPCT  scenario  in  a  specific  number 
of  runs  and  this  could  be  compared  to  the  number  of  runs  required  to  complete  the  scenario 
by  the  trainees.  This  proposed  comparison  became  problematic  early  in  the  research. 

The  availability  of  test  trainees,  especially  SMEs,  was  severely  limited  because  of  work 
schedules  of  potential  subjects.  There  were  no  SMEs  available  during  the  period  of  testing 
for  this  research.  An  even  more  limiting  factor,  discovered  in  pre-testing,  was  the  inability 
of  the  trainees  to  complete  a  successful  scenario  run  in  a  3 -hour  time  frame. 


8These  officers  are  representative  of  officers  who  might  use  the  TPCT  en  route  to 
command  of  a  PC  class  ship. 
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Two  test  subjects,  from  the  same  pool  of  available  waterfront  personnel  as  the  actual 
test  subjects  were  pre-tested  on  the  TPCT  to  collect  pilot  data.  The  pre-testing  allowed  the 
researcher  to  evaluate  the  time  required  to  conduct  each  training  session.  Because  the  TPCT 
software  requires  the  trainee  to  start  from  the  beginning  of  the  scenario  after  missing  a  critical 
decision  point,  it  increased  the  testing  time  beyond  the  expected  3  hours  and  beyond  the  time 
most  subjects  could  provide.  Trainees  were  arranged  by  SPECWARCOM  and  were  made 
available  for  a  3 -hour  training  session.  Another  difficulty  was  the  time  required  for  trainees 
to  complete  the  survey,  review  the  student  guide,  and  learn  the  basics  of  the  Integrated  Bridge 
System  (TBS).  Almost  1  hour  of  the  allotted  time  was  used  to  complete  these  aspects  of  the 
testing.  Trainees  who  had  not  completed  the  PC  Commanding  Officer  training  pipeline  were 
not  familiar  with  the  IBS  prior  to  using  the  TPCT9. 

Navy  commands  in  the  area  were  contacted  and  asked  for  volunteers.  Volunteers  were 
also  solicited  from  within  SPECWARCOM.  In  the  end,  18  test  subjects  participated  in  this 
study  from  various  commands  in  the  San  Diego,  CA  Naval  District.  Seventeen  were  U.  S. 
Navy  officers  and  one  was  a  senior  enlisted  member  from  a  PC  class  ship  with  Officer  Of  the 
Deck  (OOD)  qualification10.  Of  the  17  officers,  11  were  Lieutenants,  two  were  Warrant 
Officers,  two  were  Lieutenant  Commanders,  one  was  an  Ensign,  and  one  was  a  Commander. 
The  mean  age  of  the  group  was  3 1  years  old  ,  the  oldest  was  45,  and  the  youngest  was  24. 


9The  testing  also  revealed  that  IBS  would  not  be  known  to  the  trainees  for  whom 
the  TPCT  was  designed  because  IBS  is  not  taught  in  the  PC  commanding  Officer  training 
pipeline. 

10OOD  qualification  is  a  bridge  watch  station  that  would  require  the  member,  in 
the  absence  of  the  CO,  to  make  some  of  the  same  decisions  the  CO  would  make. 
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The  education  background,  commissioning  source,  and  ship  experience  demographics  of  the 
test  group  are  shown  in  Table  1 . 


Table  1.  TPCT  trainee  demographics. 


Trainee  Demographic  Information 

Total 

Percent  of  Group* 

Education 

BS  Engineering 

3 

17% 

BS  Non-Engineering 

11 

61% 

BA 

1 

6% 

MS  Engineering 

0 

0% 

MS  Non-Engineering 

0 

0% 

MA 

1 

6% 

Ph.D. 

0 

0% 

OTHER 

0 

0% 

Commission inq  Source 

USNA 

7 

39% 

NROTC 

6 

33% 

OCS 

1 

6% 

ECP 

1 

6% 

Seaman  to  Admiral 

0 

0% 

OTHER 

0 

0% 

Ship  Types  Served  in 

CRUDES 

12 

67% 

AMPHIB 

4 

22% 

CV 

0 

0% 

AUX 

2 

11% 

PC 

6 

33% 

MCM 

2 

11% 

OTHER 

0 

0% 

At  Sea  Experience 

COMMAND 

0 

0% 

XO 

2 

11% 

Department  Head  Jobs 

Operations 

3 

17% 

Deck 

0 

0% 

Air 

0 

0% 

Engineering 

1 

6% 

Combat  Systems 

1 

6% 

Weapons 

0 

0% 

Other 

0 

0% 

Division  Officer  Jobs 

Engineering 

15 

83% 

Deck 

5 

28% 

Air 

0 

0% 

Operations 

9 

50% 

Weapons 

1 

6% 

Combat  Systems 

0 

0% 

Other 

2 

11% 

*  All  percentages  do  not  total  100.  Trainees  may  have  more  thar 

one  degree  and 

could  have  experience  in  more 

than  one  ship  or  job  class. 

Education  and 

Commission  Source  also  have 

missing  data.  This  figure  represents  only  the 

percentage  of  the  test  subjects  with  that  particular  demographic. 
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The  instructor  during  this  study  was  a  civilian  contractor.  He  is  a  retired  U.  S.  Navy 
Commander  with  OOD  qualification  and  experience  in  the  development  of  the  PC  class  ship 
and  the  development  of  the  TPCT.  The  instructor  is  also  qualified  as  a  Navy  Master 
Instructor  and  provided  invaluable  insight  into  the  training  process  that  resulted  in  a  more 
comprehensive  review  of  the  instructor's  guide. 
B.       DATA  COLLECTION  INSTRUMENTS 

A  TPCT  training  data  package  was  developed  for  this  research.  The  complete 
package,  which  consists  of  pre-and  post-training  survey  questions,  interview  questions,  and 
instructions,  is  included  in  this  thesis.  A  group  of  10  Naval  Postgraduate  students  evaluated 
the  training  data  package  for  clarity  and  understanding  prior  to  use  in  this  research.  Field 
notes  taken  during  the  testing  and  a  report  generated  by  each  TPCT  run  were  used  in  this 
research  to  compare  each  trainee's  survey  responses.  Additionally  the  items  discussed  below, 
which  were  contained  in  the  TPCT  training  data  package  are  used  to  answer  the  research 
questions  . 

1.  Demographic  Data 

Demographic  data  were  collected  using  the  "TPCT  Trainee  Demographic  Data"  sheet 
from  the  TPCT  training  data  package  (see  Appendix  A).  In  the  originally  proposed 
methodology,  the  demographic  information  was  intended  to  provide  data  for  the  development 
of  a  model  that  would  predict  performance  in  the  simulator.  The  proposed  model  was 
patterned  after  a  model  based  on  demographic  factors  that  Cymrot  (1990)  concluded  were 
linked  to  performance  outcomes  of  naval  officers  who  qualified  as  Surface  Warfare  Officers 
(SWOs).  The  qualification  process  includes  tasks  that  resemble  some  of  the  tasking  items  in 
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the  TPCT  such  as  decision  making  during  shipboard  maneuvering.  Included  in  Cymrot's 
study  were  commissioning  source,  education,  and  shipboard  experience.  As  discussed  earlier 
in  this  thesis,  the  planned  analysis  of  performance  was  abandoned  due  to  inability  of  the 
participants  to  complete  the  scenario. 

2.  TPCT  Pre-Training  Information 

The  pre-training  information  consists  of  two  questions  developed  for  this  research  to 
examine  prior  experience  with  training  simulators  (see  Appendix  B).  Prior  experience  was 
examined  to  check  for  possible  bias  in  the  sample  group.  It  was  first  hypothesized  that  among 
S WOs  there  would  not  be  very  much  simulator  experience  and  that  positive  experience  with 
simulators  would  bias  the  trainee's  impression  of  the  TPCT  toward  more  favorable  responses. 
The  first  question  asks  if  the  trainee  has  ever  used  an  interactive  training  simulator.  Question 
two  is  only  answered  if  the  trainee  responds  positively  to  question  one.  Question  two  asks 
the  trainee  to  rate  the  prior  experience  on  a  scale  of  one  to  five,  with  negative  and  positive 
as  the  anchor  points,  and  ineffective  or  effective  as  the  anchor  points  for  the  second  part  of 
the  question.  The  last  part  of  question  two  asks  trainees  for  a  "yes"  or  "no"  answer  on 
whether  they  felt  training  simulators  were  practical  devices  for  future  use  in  the  Navy. 

3.  Post-Training  Questions 

A  set  of  nine  questions  was  developed  to  be  administered  orally  after  completion  of  the 
training  (see  Appendix  C).  The  first  five  questions  from  this  set  were  administered,  then  the 
trainee  was  given  a  battery  of  survey  questions  to  complete  (which  is  discussed  next),  then 
the  trainee  was  asked  the  remaining  four  questions  from  this  set.  The  first  five  questions  ask 
the  trainee  specific  questions  about  pressure  (not  stress)  felt  during  the  simulation,  inadequacy 
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in  the  student  guide  or  instructor's  explanations,  rceived  learning,  and  perceived  benifits 
from  the  training  experience.  The  first  three  qu  aons  of  the  remaining  four  ask  specific 
questions  regarding  levels  of  stress,  causes  of  stress,  and  attitude  toward  the  stress  caused  by 
the  trainer.  The  last  question  solicits  an  unstructured  response  by  requesting 
recommendations  and  additional  comments. 

4.  NASA  Task  Load  Index 

The  "Task  Load  Index"  ("TLX")  was  developed  at  NASA  (Hart  &  Staveland,  1988) 
to  assess  the  relative  importance  of  six  factors  in  determining  how  much  workload11  was 
experienced  during  a  task  or  exercise.  The  "NASA-TLX"  was  modified  from  its  original 
form  and  used  to  evaluate  the  task  loading  of  trainees  in  this  simulation  (see  Appendix  D). 
The  original  scale  is  a  ten-centimeter-long  line,  anchored  at  the  low  end  at  0  and  at  the  high 
end  at  one-hundred  millimeters,  scored  by  measuring  the  subject's  mark  on  the  line  to  the 
nearest  millimeter.  Professor  Baumann,  University  of  Illinois,  who  used  the  "TLX"  to 
evaluate  the  Integrated  Damage  Control  Training  Technology  (EDCTT),  recommended 
changing  the  scale  because  "measuring  things  to  the  nearest  millimeter  seems  like  over-kill 
on  a  self  report  measure  and  struck  us  as  likely  to  just  introduce  more  random  fluctuation" 
(1996b).  The  scale  was  modified  to  a  10-point  scale  anchored  at  the  low  end  at  zero  and  at 
the  high  end  at  10.  The  modified  "NASA-TLX"  contains  five  two-part  questions  and  one 
single-part  question.   The  questions  are  divided  into  the  following  task  loading  factors:  1) 


11  "Workload"  and  "task  loading"  are  generic  terms  that  describe  the  pressures, 
effort,  and  stresses,  both  mental  and  physical,  that  occur  when  a  person  is  involved  in  a 
task. 
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Mental  Demand,  2)  Physical  Demand,  3)  Temporal  Demand,  4)  Performance  Demand,  5) 
Effort  Demand,  and  6)  Frustration.  The  TPCT  was  designed  to  induce  stress  during  the 
decision  making  process  and  these  factors  are  used  to  measure  how  well  the  system 
accomplishes  this  goal.  With  the  exception  of  "physical  demand,"  a  measure  of  effectiveness 
for  the  system  is  scores  above  the  midpoint  of  "5"  on  the  scale.  The  criterion  for 
effectiveness  is  arbitrary  based  on  scores  above  5  indicating  that  the  demand  was  perceived 
as  high  rather  than  low. 

5.  User  Interface  Dimensions 

The  "User  Interface  Dimensions"  developed  by  Reeves  and  Harmon  (1994)  was  used 
as  modified  by  Johnson  (1994)  for  use  in  evaluation  of  the  IDCTT  and  further  modified  for 
use  in  this  research  (see  Appendix  E).  This  research  used  the  eight  dimensions  as  modified 
by  Johnson  and  further  changed  the  scale  of  one  to  1 1,  to  zero  to  1 0  for  ease  of  analysis.  The 
eight  dimensions  are:  1)  Ease  of  Use,  2)  Navigation,  3)  Cognitive  Load,  4)  Mapping,  5) 
Knowledge  Space  Compatibility,  6)  Information  Presentation,  7)  Media  Integration,  and  8) 
Overall  Functionality.  Each  dimension  is  addressed  as  a  separate  question  designed  to 
provide  a  measure  of  the  adequacy  of  the  human-system  interface.  A  low  score  for  any  one 
dimension  can  provide  answers  to  where  the  system  may  need  specific  improvement.  Scores 
above  "5"  on  the  scale  indicate  an  appropriate  interface.  The  criterion  was  established  based 
on  the  original  scoring  that  indicated  marks  to  the  right  of  center  were  positive. 

6.  TPCT  Post-Training  Information  Survey 

The  post-training  information  survey  was  developed  to  provide  additional  input  for 
answering  the  research  questions  in  this  thesis  (see  Appendix  F).   This  survey  contains  16 
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questions  that  ask  the  trainee  to  rate  the  system's  ease  of  use,  the  scenario's  complexity,  the 

comfort  level  of  the  environment,  usefulness  of  the  system  as  a  trainer,  effectiveness  of  some 

elements,  problems  encountered,  satisfaction,  and  recommendations.   Although  they  were 

developed  using  Johnson's  (1994)  "STUDENT  JDCTT  SURVEY"  as  guidance,  all  of  the 

questions  in  this  survey  are  specific  to  the  TPCT  system  and  its  scenario.   Each  question 

either  provides  specific  input  into  areas  of  the  trainer  that  may  need  improvement  or  provides 

an  additional  measure  of  effectiveness  based  on  trainee  satisfaction. 

C.       PROCEDURE 

An  oral  background  introduction  to  the  system  was  conducted  by  the  SIR  instructor 

and  each  trainee  was  informed  of  the  thesis  research  involved.  As  part  of  this  introduction, 

the  following  statement  was  read  to  each  trainee: 

During  this  training  period  you  will  be  assisting  in  the  beta  test  process  of  the 
TPCT.  Your  cooperation  in  completing  pre-  and  post-training  surveys  will  be 
greatly  appreciated.  The  information  gathered  in  the  surveys  is  for  research 
purposes  only,  to  assist  in  the  installation  of  the  TPCT.  In  no  way  will  your 
inputs  be  utilized  for  other  than  research.  Your  candid  comments  are  welcomed 
and  encouraged.  Your  performance  is  being  examined  for  the  purpose  of 
evaluating  the  TPCT  system. 

Prior  to  testing,  each  trainee  was  asked  to  complete  the  demographic  and  pre-test  survey 

portion  of  the  TPCT  training  data  package  in  accordance  with  the  facilitator's  guidelines 

section  of  the  training  data  package  (see  Appendix  G). 

After  the  introduction,  the  researcher  acted  solely  as  an  observer,  taking  notes  on 

comments  and  questions  from  the  trainee,  physical  actions  of  the  trainee,  interactions  between 

the  trainee  and  the  instructor,  and  responses  by  the  instructor  to  the  trainee's  questions.  Each 

trainee  was  allotted  a  15-30  minute  period  to  review  the  student  guide  and  ask  questions.  A 
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complete  introduction  was  then  given  to  the  student  by  the  instructor  that  covered  operating 
the  system,  operating  the  IBS,  and  what  role  the  instructor  would  play. 

The  instructor's  role  changed  after  two  trainees  when  it  was  realized  that  more 
guidance  was  needed  to  understand  key  operating  features  of  the  system.  The  most  prevalent 
change  was  the  inclusion  of  a  more  detailed  brief  on  the  IBS  and  instructor  assistance  in  this 
area.  From  this  point  on,  the  instructor  operated  the  IBS  for  the  student  if  necessary.  If  the 
trainee  made  a  decision  that  required  the  use  of  the  IBS,  recognized  that  IBS  was  needed,  and 
did  not  know  how  to  operate  the  IBS  to  implement  the  decision,  the  instructor  could  be  asked 
for  assistance.  The  instructor  would  only  guide  the  trainee  on  the  operating  procedure 
required  to  implement  the  decision,  not  in  the  decision  making  process.  The  trainee  could  ask 
questions,  but  the  instructor  refrained  from  answering  them  as  much  as  possible  to  maintain 
the  autonomous  decision  making  situation. 

Each  trainee  participated  in  the  scenario  until  a  decision  point  was  missed,  and  the 
system  indicated  an  error.  The  instructor  would  then  debrief  the  mission  to  the  point  of  the 
error,  explaining  and  discussing  the  decision  and  training  points  to  this  point  in  time.  The 
trainee  could  also  ask  system  operation  questions  and  comment  freely  on  the  system.  After 
completion  of  the  debrief,  the  scenario  was  restarted  from  the  beginning,  and  the  trainee 
would  again  proceed  until  a  decision  point  was  missed.  Each  trainee  was  usually  only  able 
to  participate  in  the  scenario  twice,  at  the  most  making  it  half-way  through  the  scenario  on 
the  second  try,  during  the  allotted  3 -hour  training  session. 

After  completion  of  the  debrief  for  the  last  scenario  try,  the  trainee  was  asked  to 
complete  the  post-training  evaluation.  The  post-training  phase  consisted  of  five  structured 
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interview  questions  followed  by  completion  of  the  "Task  Load  Index,"  "User  Interface 
Dimensions",  and  TPCT  Post-Training  Information  survey  portion  of  the  TPCT  training  data 
package.  The  researcher  was  available  to  answer  questions  concerning  the  survey  during  this 
time.  After  completing  the  survey,  the  trainee  was  then  interviewed  with  four  more  questions, 
the  last  being  an  open-ended  request  for  recommendations  and  comments. 

At  the  completion  of  each  training  session,  the  researcher  and  the  instructor  discussed 
the  session  and  recommendations  that  were  made.  The  TPCT  also  generated  a  report  for 
each  session  on  the  actions  taken  by  trainee.  The  report  indicated  Decision  Points  reached 
and  actions  completed  by  the  trainee.  The  results  of  the  observation,  survey  results, 
interviews,  and  scenario  reports  provide  the  data  for  the  formative  evaluation  of  the  TPCT. 
The  formative  evaluation  of  the  data  are  presented  and  discussed  in  the  next  chapter. 
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V.  DATA  PRESENTATION  AND  ANALYSIS 

This  chapter  presents  the  data  obtained  using  the  Tactical  Patrol  Craft  Trainer  (TPCT) 
training  data  package,12  the  researcher's  observations,  and  the  trainees'  comments.  The  data 
are  presented  as  frequency  distributions  for  the  quantitative  elements  of  the  Pre-Training 
survey,  CCNASA  Task  Load  Index"  ("TLX"),  "User  Interface  Dimensions,"  and  TPCT  Post- 
Training  Information  survey.  Additionally  the  statistical  mean  and  standard  deviation  are 
presented  as  appropriate. 

Because  of  their  qualitative  nature,  the  observations  and  comments  are  summarized 
based  on  significance  and  frequency.  The  significance  of  the  observations  or  comments  are 
judged  subjectively  by  the  researcher  and  influenced  by  the  research  questions  outlined  in  this 
thesis.  Frequency  is  not  a  criteria  for  inclusion,  but  it  does  contribute  to  significance. 
Comments  from  trainees  are  summarized. 
A.         TPCT  PRE-TRAINING  INFORMATION 

The  Pre-Training  Information  survey  (see  Appendix  B)  was  developed  to  determine 
the  computer-based  simulator  experience  level  of  the  trainees  and  examine  any  preconceived 
opinions  about  this  technology  prior  to  introduction  to  the  TPCT.  Question  #1  asked  the 
trainees  if  they  had  ever  used  an  interactive  training  simulator  (yes  or  no).  Ten  of  18  (56 
percent)  of  the  trainees  indicated  that  they  had  prior  simulator  experience.  The  trainees  that 
answered  yes  to  question  #1  were  asked  to  answer  the  three  parts  of  question  #2.  Question 


12The  contents  of  this  package  are  detailed  in  Chapter  IV  and  included  in  this  thesis 
as  Appendices  A  thru  G. 
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#2A  asked  the  trainees  to  rate  their  previous  experience  as  negative  or  positive.  These  ten 
trainees  rated  their  experience  as  positive  with  a  mean  score  of  4.5  with  a  standard  deviation 
of  0.71  on  a  scale  of  1  to  5.  Question  #2B  asked  the  trainees  to  rate  the  previous  training  as 
ineffective  or  effective.  These  ten  trainees  rated  the  training  as  effective  with  a  mean  score 
of  4. 1  with  a  standard  deviation  of  0.57  on  a  scale  of  1  to  5.  One-hundred  percent  of  the 
group  with  prior  experience  also  answered  "yes"  to  question  #2C,  indicating  that  they 
envisioned  future  applications  for  computer-based  simulator  training  in  the  Navy.  The 
number  of  trainees  in  the  TPCT  test  group  with  prior  simulator  experience  was  higher  than 
expected,  and  the  experiences  were  positive.  This  finding  may  bias  the  results  of  the  research 
because  trainees  who  enjoy  the  type  of  training  they  are  involved  in  are  more  likely  to  respond 
favorably  to  questions  about  the  effectiveness  of  the  training  (Ellis,  1986). 
B.         TASK  LOAD  INDEX  EVALUATION 

The  "TLX"  developed  by  NASA  and  modified  for  this  research  is  divided  into  six 
factors  (see  Appendix  D).  These  factors  are  used  in  evaluating  the  TPCT's  ability  to  impose 
high  levels  of  various  cognitive  demands  that  contribute  to  stress  during  the  decision  making 
process.  The  data  for  each  factor  are  presented  and  discussed  separately. 

Mental  Demand  experienced  by  the  trainees  is  evaluated  using  data  from  the  two  parts 
of  question  #1  as  displayed  in  Figure  5.  The  first  part  (A)  identifies  how  much  mental  demand 
was  required.  The  mean  score  was  7.78  with  a  standard  deviation  of  1 .35.  The  second  part 
(B)  identifies  the  degree  of  difficulty  in  mental  demand.  The  mean  score  was  7.1 1  with  a 
standard  deviation  of  1.32.  The  trainees  responded  that  the  task  required  high  levels  of  mental 
activity  and  was  demanding. 
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Figure  5.  Task  Load  Index  questions  1 A  and  IB  percentages. 


Physical  Demand  experienced  by  the  trainees  is  evaluated  using  data  from  the  two 
parts  of  question  #2  as  displayed  in  Figure  6.  The  first  part  (A)  identifies  how  much  physical 
activity  was  required.  The  mean  score  was  1.28  with  a  standard  deviation  of  1.23.  The 
second  part  (B)  identifies  the  difficulty  of  physical  activity.  The  mean  score  was  3.89  with 
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Figure  6.  Task  Load  Index  questions  2A  and  2B  percentages. 
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a  standard  deviation  of  1.75.  The  trainees  respond  that  the  task  required  very  little  physical 
activity.  But,  even  though  the  data  show  the  physical  activity  was  not  difficult,  it  should  be 
noted  that  over  40  percent  of  the  respondents  scored  the  difficulty  at  or  above  the  set 
criterion  of  5.13  This  rating  may  be  caused  by  an  observed  variance  in  the  trainee's  ability  to 
easily  operate  the  touchscreen  monitor  (the  only  physical  aspect  of  the  trainer). 

Temporal  Demand  experienced  by  the  trainees  is  evaluated  using  data  from  the  two 
parts  of  question  #3  as  displayed  in  Figure  7.  The  first  part  (A)  identifies  how  much  time 
pressure  the  trainee  felt  due  to  the  pace  of  the  task.  The  mean  score  was  6.22  with  a  standard 
deviation  of  2.05.  The  second  part  (B)  identifies  whether  the  trainee  viewed  the  pace  of  the 
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Figure  7.  Task  Load  Index  questions  3A  and  3B  percentages. 


13  The  criterion  for  evaluating  survey  instruments  scaled  0  to  10  was  established  in 
Chapter  III.     Low  and  high  are  determined  relative  to  the  midpoint  of  5  on  the  scale. 
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task  as  slow  or  rapid.  The  mean  score  was  6.28  with  a  standard  deviation  of  1 .27.  Although 
both  parts  indicate  high  temporal  demand,  variation  in  the  amount  of  time  pressure  felt  is 
distinctive  because  the  TPCT  scenario  was  designed  to  create  a  highly  time-sensitive 
environment.  The  data  show  that  the  pace  is  rapid  with  most  responses  occurring  at  or  above 
a  score  of  6,  but  the  resulting  pressure  experienced  by  the  trainees  varied  considerably  with 
over  30  percent  responding  at  or  below  the  criterion  of  5. 

Performance  Demand  experienced  by  the  trainees  is  evaluated  using  data  from  the  two 
parts  of  question  #4  as  displayed  in  Figure  8.  The  first  part  (A)  indicates  how  successful  the 
trainee  felt  in  the  accomplishment  of  the  tasks.  The  mean  score  was  6.28  with  a  standard 
deviation  of  2.47.  The  second  part  (B)  identifies  how  satisfied  the  trainee  was  with  individual 
performance.  The  mean  score  was  4.67  with  a  standard  deviation  of  2.68.  The  data  indicate 
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Figure  8.  Task  Load  Index  questions  4 A  and  4B  percentages. 
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that  the  trainees  generally  felt  more  successful  than  they  felt  satisfied.  However,  there  are 
large  variances  in  the  responses  to  both  questions.  The  inability,  due  to  time  restrictions,  to 
complete  the  scenario  may  have  impacted  this  factor.  The  trainees  may  have  felt  some 
success  from  advancing  beyond  some  difficult  task  in  the  scenario,  yet  satisfaction  could  be 
diminished  because  the  trainee  could  not  complete  the  entire  scenario. 

Effort  Demand  experienced  by  the  trainees  is  evaluated  using  data  from  the  two  parts 
of  question  #5  as  displayed  in  Figure  9.  The  first  part  (A)  identifies  how  much  mental  effort 
was  required  to  achieve  the  trainee's  level  of  performance.  The  mean  score  was  6.5  with  a 
standard  deviation  of  1.34.  The  second  part  (B)  identifies  how  much  physical  effort  was 
required  to  achieve  the  trainee's  level  of  performance.  The  mean  score  was  1.78  with  a 
standard  deviation  of  2.26.  The  data  indicate  that  on  average,  the  mental  effort  required  to 
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Figure  9.  Task  Load  Index  questions  5  A  and  5B  percentages. 
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accomplish  a  level  of  performance  was  significantly  higher  than  the  physical  effort.  But  notice 
should  also  be  given  to  the  several  outliers  who  appeared  to  have  some  physical  difficulties 
in  performance  achievement.  The  mental  effort  is  expected  in  a  computer-based  simulator, 
but  physical  effort  was  not  expected  to  be  a  factor.  Difficulty  with  the  touchscreen  was 
reported  by  some  trainees  and  may  have  contributed  to  this  finding. 

Frustration  experienced  by  the  trainees  is  evaluated  using  data  from  question  #6  as 
illustrated  in  Figure  10.  The  data  indicate  levels  of  frustration  the  trainees  felt  during  the 
simulation.  The  mean  score  was  5.3  with  a  standard  deviation  of  1 .94  .  Although  the  mean 
score  is  above  the  criterion  of  5,  the  majority  of  responses  are  grouped  around  this  criteria 
indicating  that  the  frustration  levels  were  not  extremely  high.  Some  trainees  expressed 
specific  frustration  at  not  being  able  to  finish  the  scenario.  Other  factors  that  could  contribute 
to  frustration  are  lack  of  experience  with  the  Integrated  Bridge  System  (IBS)  or  problems 
interfacing  with  the  system. 


Figure  10.  Task  Load  Index  question  #6  response  percentages. 
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The  cumulative  results  of  the  "TLX"  indicate  that  the  TPCT  induces  high  mental 
demand  and  requires  low  levels  of  physical  activity.  The  temporal  demand  factor  is  rated 
high,  but  the  time  pressure  aspect  is  lower  than  expected.  Frustration  levels  are  within  a 
reasonable  range  and  should  not  be  considered  detrimental  to  the  training  process.  The  TPCT 
system  appears  to  be  effective  in  generating  the  intended  stress  levels  of  trainees. 
C.         USER  INTERFACE  DIMENSIONS 

The  "User  Interface  Dimensions"  (see  Appendix  E)  are  used  in  this  research  to 
determine  a  measure  of  the  adequacy  of  the  human-system  interface.  Each  scale  is  anchored 
by  a  negative  evaluation  of  the  interface  (e.g.,  "Difficult  to  Use")  at  the  low  end,  and  a 
positive  evaluation  at  the  high  end  of  the  scale  (e.g.,  "Easy  to  Use").  The  mean  score  of 
trainees'  responses  to  all  eight  dimensions  are  shown  in  Figure  1 1 . 
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All  dimensions  of  the  TPCT  were  evaluated  above  the  criterion  of  "5"  for 
consideration  as  an  adequate  human-system  interface.  The  "Ease  of  Use"  dimension  had  a 
mean  score  of  6.94  with  a  standard  deviation  of  1 .35.  The  "Navigation"  dimension  had  a 
mean  score  of  6.61  with  a  standard  deviation  of  2.06.  The  "Cognitive  Load"  dimension  had 
a  mean  score  of  7.67  with  a  standard  deviation  of  1.61.  The  "Mapping"  dimension  had  a 
mean  score  of  7.06  with  a  standard  deviation  of  2.07.  The  "Knowledge  Space  Compatibility" 
dimension  had  a  mean  score  of  7.50  with  a  standard  deviation  of  1.50.  The  "Information 
Presentation"  dimension  had  a  mean  score  of  7.61  with  a  standard  deviation  of  1.72.  The 
"Media  Integration"  dimension  had  a  mean  score  of  7.61  with  a  standard  deviation  of  1.04. 
The  "Overall  Functionality"  dimension  had  a  mean  score  of  8.44  with  a  standard  deviation 
of  1.15.  Dimensions  1,3,  and  5  thru  8  were  all  positive  with  no  indications  of  failure  in  the 
human- system  interface,  but  the  data  for  dimensions  2  (Navigation)  and  4  (Mapping) 
indicated  possible  problems. 

Although  "Navigation"  was  not  below  the  set  criterion,  it  was  lower  than  the  rest  of 
the  dimensions.  Over  20  percent  responded  at  or  below  the  criterion  of  5,  with  two  trainees 
responding  as  low  as  3  on  the  scale.  The  trainees  expressed  concern  that  they  were  unfamiliar 
with  the  system  and  had  to  find  things  "on  the  fly."  Suggestions  were  made  to  allow  the 
trainee  more  time  to  review  the  student  guide  and  allow  the  main  screen  to  be  accessible  prior 
to  starting  the  scenario  for  the  trainee  to  get  familiar  with  the  buttons,  their  functions  and 
contents.  The  low  score  in  the  "Navigation"  dimension  relative  to  other  dimensions  supports 
recommendations  made  by  trainees  to  provide  the  trainees  more  familiarization  with  the 
interface  before  use. 


47 


Although  the  "Mapping"  dimension's  mean  was  well  above  the  criterion  of  5,  over 
40  percent  of  the  trainees  responded  at  or  below  the  criterion.  This  variation  in  the  dimension 
may  be  explained  in  two  different  ways.  Two  trainees  asked  for  clarification  of  the  dimension 
while  taking  the  survey.  This  dimension  was  also  questioned  during  the  survey  review 
conducted  by  Naval  Postgraduate  students.  Confusion  on  this  question  may  have  added  to 
its  variation,  but  the  fifth  dimension  was  also  questioned  and  required  clarification  to  some 
of  the  trainees  and  it  did  not  have  the  same  large  variation.  Some  trainees  expressed  a  feeling 
of  being  "lost  in  the  scenario,"  of  "not  knowing  that  I  was  behind  schedule."  The  ability  to 
follow  the  progress  and  keep  to  a  rigid  time  line  is  a  major  training  point  in  the  system  and 
some  trainees  may  have  evaluated  the  "Mapping"  dimension  somewhat  low  because  they  felt 
there  should  be  more  clues  when  they  were  falling  behind.  But  the  trainees  also  expressed 
that  this  was  not  a  bad  aspect  of  the  trainer  because  it  reinforced  ideas  of  "keeping  an  eye  on 
the  big  picture." 

Even  with  possible  improvements  indicated  by  the  "Mapping"  and  "Navigation" 
dimensions,  the  interface  was  perceived  as  appropriate.   The  cumulative  evaluation  of  the 
TPCT,  using  the  "User  Interface  Dimensions,"  is  that  it  is  easy  to  use  and  the  trainees  felt 
the  media  integration  and  overall  functionality  were  good. 
D.         TPCT  POST-TRAINING  INFORMATION  SURVEY 

The  post-training  information  survey  (see  Appendix  F)  results  are  consistent  with  the 
previous  survey  instruments  and  provide  additional  input  for  answering  the  research  questions 
in  this  thesis.  Each  question  is  discussed  in  the  following  paragraphs. 

Question  1  asks  for  the  trainee's  perception  of  the  TPCT  system's  ease-of-use.  The 


48 


average  score  was  3.9  with  a  standard  deviation  of  0.9  on  a  1  to  5  scale.  The  results  indicate 
that  the  majority  of  trainees  perceived  the  trainer  as  easy  to  use.  This  question  specifically 
validated  the  results  of  the  "User  Interface  Dimensions"  ease-of-use  dimension  and  indicated 
that  the  system,  as  a  whole,  was  easy  to  use  and  thus  the  interface  was  not  a  major  problem. 

Question  2  looked  more  specifically  at  the  touchscreen  interface  to  rate  the  trainee's 
perception  of  its  ease-of-use.  This  was  examined  because  Johnson  (1994)  found  that  the 
IDCTT  touchscreen  had  been  a  problem.  The  average  score  for  the  TPCT  was  4. 17  with  a 
standard  deviation  of  0.7  on  a  1  to  5  scale.  The  results  indicate  that  the  trainees  considered 
the  touchscreen  easy  to  use.  Although  earlier  results  indicate  possible  problems  with  the 
touchscreen,  the  results  here  indicate  that  the  difficulties  experienced  by  the  trainees  were  not 
severe. 

Question  3  was  intended  to  evaluate  the  trainee's  perception  of  the  scenario  instead 
of  the  system  or  interface.  The  results  of  this  question's  four  parts  are  shown  in  Table  2. 

Table  2.  Post-training  information  survey  question  3  results. 


TPCT  SCENARIO 



SCALE 

AVERAGE 

Easy  to  Difficult 

3.4 

Confusing  to  Understandable 

4.0 

Slow  to  Fast 

3.5 

Unrealistic  to  Realistic 

3.9 

Scale:  1  to  5 
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The  scenario  was  designed  to  be  understandable  yet  difficult,  fast,  and  realistic.  The  average 
score  for  each  part  to  this  question  was  above  the  midpoint  of  3  and  indicates  that  the 
scenario  was  understandable  while  remaining  difficult,  fast,  and  realistic. 

Question  4  looks  at  environmental  factors  that  may  have  contributed  to  the  trainee's 
perception  of  the  trainer.  Because  the  trainer  was  located  in  a  temporary  area  that  may  have 
proved  distracting,  the  trainees  were  asked  to  rate  the  temperature,  comfort  level,  and  noise 
level.  The  trainees  rated  the  temperature  neither  too  cold  nor  too  hot  with  an  average  score 
of  3.0  with  a  standard  deviation  of  0.34  on  a  scale  of  1  to  5.  Comfort  level  was  scored 
toward  "Very  Comfortable"  by  the  trainees  with  an  average  score  of  3.83  with  a  standard 
deviation  of  0.62  on  a  scale  of  1  to  5.  The  noise  level  did  not  seem  to  disturb  the  trainees 
who  indicated  that  it  was  neither  too  quiet  nor  too  loud  with  an  average  score  of  3.0  with  a 
standard  deviation  of  0.34  on  a  scale  of  1  to  5.  So,  the  trainees  felt  comfortable  and  the 
environment  should  not  have  affected  their  perceptions  of  the  system. 

The  TPCT  system  was  designed  as  a  trainer  for  the  prospective  PC  class  Commanding 
Officers  and  the  trainees  were  asked  in  question  5  about  the  utility  of  the  system  to  this  end. 
Did  the  trainees  think  that  the  system  was  an  effective  trainer?  The  response  to  this  question 
was  very  positive  with  an  average  score  of  4.72  with  a  0.46  standard  deviation  on  a  scale  of 
1  to  5 .  The  trainees  also  responded  favorably  to  the  question  (number  6),  which  asked  if 
there  was  utility  for  this  type  of  system  in  other  decision  making  training.  The  response  to 
question  6  was  an  average  score  of  4.56  with  a  0.70  standard  deviation  on  a  scale  of  1  to  5. 
From  these  two  questions,  it  appears  that  the  trainees  perceived  the  training  as  constructive 
despite  any  specific  problems  with  the  system. 
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Question  7  looks  at  some  of  the  specific  problems  indicated  in  the  literature  review 
that  may  cause  the  system  to  appear  ineffective  to  the  trainees.  The  numbers  of  trainees  that 
indicated  difficulty  with  a  specific  aspect  of  the  system  are  displayed  in  Table  3 .  Although  the 
trainees  indicated  in  other  survey  questions  that  the  touchscreen  was  easy  to  use,  five  trainees 
expressed  some  difficulty  with  the  technology.   The  most  common  comment  was  that  the 

Table  3.  Post-training  information  survey  question  7  results. 


TRAINEE  DIFFICULTIES 

Number  of  trainees  who  experienced  difficulty  with: 

Inputting  information  into  the  touchscreen  monitor 

Understanding  audio  reports 

Speed  or  volume  of  the  information  presented 

Inputting  information  into  the  Integrated  Bridge  System  (IBS) 

Other* 


Answers  to  "Other"  are  explained  in  the  text. 


TOTAL 

PERCENT 

5 

28 

3 

17 

0 

0 

7 

39 

2 

11 

n  = 

18 

trainee  was  not  familiar  with  touchscreens  and  it  "took  some  getting  used  to."  Three  trainees 
indicated  that  they  had  difficulty  understanding  audio  reports,  but  the  cause  was  explained  as 
"being  involved  in  something  else"  or  "not  paying  attention."  None  of  the  trainees  had 
difficulty  with  the  speed  or  volume  of  information  presented.  The  most  common  difficulty, 
as  expected  from  the  pre-testing,  was  inputting  information  into  the  Integrated  Bridge  System 
(IBS).  Seven  trainees  expressed  that  they  were  not  familiar  with  the  IBS  and  this  made  the 


51 


entire  training  more  difficult.  Other  difficulties  expressed  included  "visualizing  what  was 
going  on  outside  the  ship"  and  "dealing  with  the  system  warping  when  I  wasn't  ready."14 

Question  8  asks  the  trainees  how  effective  the  student  guide  was  in  preparing  them 
for  the  training.  The  response  to  this  question  was  positive  with  an  average  score  of  3.89 
with  a  standard  deviation  of  0.58  on  a  scale  of  1  to  5.  This  is  high  considering  there  were 
numerous  suggestions  made  to  allow  the  trainee  more  time  to  study  the  student  guide.  One 
trainee  was  asked  about  this  contradiction  and  commented  that  the  guide  was  good,  but  more 
time  was  needed  to  digest  the  information.  Although  more  time  may  be  needed  with  the 
student  guide  prior  to  using  the  system,  the  student  guide  appears  to  be  an  appropriate  pre- 
training  instrument.  Suggestions  for  improvements  to  the  student  guide  are  discussed  later 
in  this  chapter. 

Questions  9  and  10  rate  the  trainee's  perception  of  instructor  effectiveness  and 
amount  of  information  provided  during  the  scenario.  The  trainees  appeared  to  be  very 
satisfied  with  the  information  provided  by  the  instructor  prior  to  operating  the  system. 
Question  9  had  an  average  score  of  4.44  with  a  standard  deviation  of  0.51  on  a  scale  of  1  to 
5.  Instructor  intervention  did  not  seem  to  disturb  the  trainees  who  indicated  in  response  to 
question  10  that  the  instructor  provided  neither  not  enough  nor  too  much  information  with 
an  average  score  of  3.22  with  a  standard  deviation  of  0.55  on  a  scale  of  1  to  5.  So,  the 
trainees  felt  comfortable  with  the  information  provided  by  the  instructor  prior  to  and  during 
the  scenario. 


14The  simulation  is  shortened  through  the  use  of  "warping,"  which  advances  the 
trainee  and  ship's  position  in  the  mission  at  an  accelerated  rate.  During  this  time  no 
actions  are  taken  by  the  trainee. 
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Question  1 1  rates  the  trainees'  feelings  of  success  during  the  scenario.  The 
expectation  was  that  either  the  trainees  would  not  feel  successful  because  they  did  not 
complete  the  scenario,  or  they  might  feel  successful  based  on  what  they  may  have  learned. 
The  responses  to  this  question  indicate  that  the  trainees  did  not  feel  very  successful  with  an 
average  score  of  2.44  with  a  1.10  standard  deviation  on  a  scale  of  1  to  5.  Individual 
comments  from  trainees  support  the  assertion  that  the  data  representing  perception  of  success 
are  correlated  with  completion  of  the  scenario. 

Question  12  asked  for  input  on  problems  encountered  while  using  the  TPCT.  The 
following  list  represents  a  summary  of  the  trainees'  specific  problems: 

•  Warping  compounded  mistakes  and  created  confusion. 

•  Trainees  could  not  "ask"  the  computer  for  recommendations. 

•  They  were  unfamiliar  with  the  touchscreen. 

•  They  were  unfamiliar  with  the  IBS. 

•  The  IBS  voyage  planner  stopped  operating  properly. 

The  problem  with  the  IBS  voyage  planner  was  software  related  and  corrected  immediately. 
The  perceived  difficulty  with  the  touchscreen  resulted  in  a  recommendation  for  the  addition 
of  a  "demo  button"  on  the  trainer  screen  to  allow  practice  on  the  user  interface  without 
activating  the  scenario.  The  difficulty  with  the  DBS  and  the  perceived  need  to  verbally 
interface  with  the  system  are  discussed  in  further  detail  later  in  this  chapter.  The  warp 
function  was  determined  to  be  correctable  if  the  trainee  is  indoctrinated  into  the  procedure 
for  making  adjustments  while  warping.  The  system  will  stop  warping  if  the  trainee  initiates 
an  action,  i.e.,  orders  a  course  or  speed  change. 
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Question  13  asked  for  input  on  what  aspects  of  the  TPCT  the  trainees  liked  the  most. 
The  following  list  represents  a  summary  of  the  trainees'  responses: 

Realism 

Great  graphics 

Detail 

Audio  reports 

Time  pressure 

Touchscreen 

Ten  trainees  responded  that  they  really  liked  the  realism  of  the  system.  The  graphics,  detail, 
and  audio  are  integral  to  the  high  fidelity  system  and  essential  to  the  perception  of  reality. 
The  pressure  of  time  management  was  liked  by  the  trainees  even  though  this  was  determined 
by  the  "TLX"  to  be  a  stressor.  The  trainees  indicated  that  the  time  pressure  was  a  realistic 
aspect  of  ship  board  operations. 

Question  14  asked  for  input  on  what  aspects  of  the  TPCT  the  trainees  liked  the  least. 
The  following  list  represents  a  summary  of  the  trainees'  responses: 

Blank  video  screen,  lack  of  video  at  times 

Dead  time 

Glare  on  the  trainer  monitor 

Not  being  able  to  ask  watchstanders  questions 

Uncontrollable  warping 
The  most  common  dislike  was  the  absence  of  video  when  not  receiving  reports  from  the 
system.    The  trainees  indicated  that  this  detracted  from  the  reality  and  recommended  a 
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continuos  video  stream.  Additionally,  the  trainees  indicated  that  video  could  be  used  to 
provide  visual  cues  to  aid  in  the  decision  making  process,  i.e.,  video  of  a  fog  bank  may 
influence  the  decision  to  slow  the  ship  or  not.  The  trainees  did  not  like  the  "dead  time" 
between  events.  This  appeared  to  cause  anticipation  stress,  and  it  was  not  determined  if  this 
stress  was  beneficial  or  a  hindrance  to  the  goal  of  the  system.  The  glare  on  the  trainer 
monitor  was  noted  by  several  trainees,  and  it  was  recommended  that  the  trainer  monitor  be 
changed  to  an  etched  finished  monitor  similar  to  the  IBS  monitor.  The  verbal  interaction  with 
watchstanders  is  discussed  later  in  this  chapter. 

Question  15  asked  for  specific  recommendations  for  future  use  of  the  TPCT.  The 
following  list  represents  a  summary  of  the  trainees'  recommendations: 

•  Training  for  all  PC-bound  officers 

•  Ship  handling  scenarios,  i.e.,  anchoring  evolutions,  refueling,  and  man 
overboard 

•  More  mission-related  scenarios 

•  Casualty  control  scenarios 

•  Watch  team  training  and  training  for  other  watch  stations 

Question  16  asked  for  specific  recommendations  for  future  use  of  the  simulation 
technology  used  in  the  TPCT.  The  following  list  represents  a  summary  of  the  trainees' 
recommendations : 

•  Engineering  Casualty  Control  (ECC)  drills 

•  Damage  control  training 

•  Similar  training  for  different  ship  types 

The  post-training  information  survey  data  indicate  that  the  TPCT  is  easy  to  use,  liked 
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by  the  trainees,  and  realistic.   But,  the  survey  also  indicated  some  improvement  areas  and 
resulted  in  several  recommendations.  The  survey  is  further  supported  by  information  gained 
through  observation  and  trainees'  comments. 
E.         OBSERVATIONS  AND  ADDITIONAL  TRAINEE  COMMENTS 

During  each  training  session  the  trainees  were  observed  and  notes  were  taken  on  their 
comments  and  questions,  physical  actions  of  the  trainee,  interactions  between  the  trainee  and 
the  instructor,  and  responses  by  the  instructor  to  the  trainee's  questions.  This  section 
describes  those  observations  as  they  apply  to  the  research  questions  and  compare  to  other 
data  instruments. 

The  "TLX"  data  show  that  the  TPCT  induces  stress  as  intended.  Observation  of  the 
trainees  also  indicated  that  stress  was  created  by  the  system.  The  trainees  would  fidget,  wring 
their  hands,  and  hesitate  to  make  some  decisions.  Although  the  trainees  would  not  admit  to 
being  stressed,  they  did  indicate  that  they  felt  "pressure"  to  make  the  right  decisions  and 
maintain  the  time  line.  They  also  indicated  pressure  as  a  result  of  the  fast  pace  as  well  as 
unfamiliarity  with  various  components. 

At  the  start  of  each  training  session,  the  touchscreen  monitor  appeared  to  create  stress 
in  the  trainees.  But  the  trainees  were  observed  becoming  more  comfortable  with  the 
technology  by  the  start  of  the  second  scenario  run.  Most  trainees  even  seemed  to  enjoy  the 
use  of  the  touchscreen.  This  observation  combined  with  the  data  from  the  post-training 
survey  supports  a  recommendation  made  by  the  trainees  to  alter  the  system  to  allow  practice 
on  the  touchscreen  to  build  familiarity.  Recommendations  were  also  made  to  investigate 
voice  recognition  as  an  alternate  means  of  interfacing  with  the  system. 
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The  trainees  were  observed  "talking"  to  the  system.  Most  often,  the  trainees  would 
say  "no"  out  loud  to  deny  a  recommended  action.  If  the  system  recommended  a  course 
change  and  the  trainee  did  not  agree,  then  no  action  on  the  part  of  the  trainee  was  required 
to  deny  the  request,  but  the  trainees  expressed  a  need  to  actually  do  something  either  verbally 
or  physically  to  indicate  a  "no"  response. 

The  trainees  were  also  observed  to  constantly  ask  for  information  from  the  instructor 
to  assist  them  in  the  decision  making  process.  As  an  example,  one  trainee  asked  if  the  ship 
had  departed  enemy  waters  yet.  This  is  information  that  the  trainee  should  be  able  to  find  or 
would  have  known  if  attention  was  paid  to  audio  reports.  If  the  instructor  answered  these 
questions,  the  scenario  would  become  too  easy.  The  instructor  for  this  evaluation  usually 
responded  to  these  questions  with  "I  can  not  give  you  that  information."  After  discussing  this 
with  the  instructor,  it  was  decided  that  to  simulate  reality  maybe  the  instructor  could  answer 
as  a  crew  member  might  with  something  along  the  lines  of  "I'll  get  back  to  you  on  that,  sir." 
The  instructor  recommended  clarifying  this  in  the  instructor's  guide.  But,  it  should  be  noted 
that  the  data  show  that  the  trainees  were  happy  with  the  information  provided. 

The  instructor's  guide  also  did  not  include  specific  information  on  conducting  the  pre- 
training  brief.  Although  the  instructor  for  this  evaluation  conducted  excellent  briefs  and  the 
trainees  were  satisfied  with  the  information  presented,  discussions  with  the  instructor  resulted 
in  a  recommendation  that  the  brief  be  standardized  and  included  in  the  instructor's  guide. 

The  pre-training  brief  for  this  evaluation  included  an  extensive  review  of  the  IBS  that 
could  almost  be  considered  training  in  itself.  The  IBS  is  a  complicated  navigation  and  control 
system  that  requires  some  detailed  instruction  and  practice  to  master.  Even  with  this  brief, 
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the  trainees  experienced  problems  operating  the  IBS  mostly  because  they  were  unfamiliar  with 
the  system.  The  trainees  recommended  being  taught  the  IBS  before  using  the  TPCT. 

The  trainees  noted  the  following  items  that  detracted  from  the  reality  of  the  system. 
At  one  point  in  the  scenario  the  fuel  percentage  indicated  2  percent  fuel  remaining.  It  was 
noted  by  the  trainees  that  the  PC  class  ship  will  lose  fuel  suction  and  fail  to  operate  if  the 
percentage  drops  below  10  percent,  so  2  percent  and  still  operating  is  not  realistic.  In  the 
event  of  loss  of  steering  gear,  the  PC  class  ship  has  the  capability  of  "steering  with  engines" 
while  this  is  not  an  option  in  the  TPCT.  The  TPCT  has  static  gauges  that  do  not  change  in 
the  case  of  an  engine  casualty.  It  was  recommended  by  trainees  that  the  gauges  be 
programmed  to  indicate  actual  casualty  conditions.  Reality  was  also  questioned  when  it  was 
observed  that  the  successful  completion  of  a  decision  point  only  required  a  minimal  decrease 
in  speed  instead  of  a  decrease  in  speed  according  to  standard  ship  board  procedure. 

The  trainees  kept  asking  for  a  device  known  as  a  "whiz  wheel."  The  small  plastic 
hand-held  device  is  used  on  ships  to  calculate  time-distance  problems.  Although  the  time- 
distance  problems  can  be  calculated  by  hand,  it  was  recommended  that  a  "whiz  wheel"  be 
made  available  for  the  trainees  to  use.  Additionally,  the  trainees  did  not  easily  understand 
how  to  utilize  the  IBS  system  in  conjunction  with  the  clock  to  ensure  they  were  on  schedule. 

The  trainees  appeared  somewhat  agitated  that  they  could  not  finish  the  scenario  in  the 
3 -hour  time  frame.  The  instructor  and  trainees  recommended  that  the  scenario  include  an 
alternate  starting  point  to  reduce  the  time  involved  in  starting  over.  But  it  was  also  noted  that 
the  trainees  expressed  a  desire  to  return  and  try  the  system  again,  especially  if  more  and  varied 
scenarios  were  added. 
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The  observation  of  training  resulted  in  qualitative  support  for  earlier  findings  that  the 
TPCT  induces  stress  as  intended,  the  trainees  like  the  system,  it  is  relatively  easy  to  use,  and 
trainees  perceive  the  training  as  beneficial.  This  phase  of  the  evaluation  also  resulted  in 
recommendations  for  improvements  to  the  system.  The  following  chapter  summarizes  the 
findings  and  presents  recommendations  based  on  the  data. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 

The  purpose  of  this  chapter  is  to  present  the  results  of  the  formative  evaluation  of  the 
Tactical  Patrol  Craft  Trainer  (TPCT)  with  regard  to  the  research  questions  presented  in  the 
introduction.  The  literature  review  indicated  that  formative  evaluation  is  important  to  the 
development  of  a  successful  training  system,  and  prior  research  provided  insight  on  specific 
evaluation  tools  to  use  and  an  indication  of  expected  results.  The  methodology  used  in  this 
research  was  directed  at  answering  specific  research  question,  the  results  of  which  are 
presented  in  the  following  sections. 
A.         RESEARCH  QUESTIONS 

This  research  was  based  on  the  four  questions  presented  in  chapter  1  of  this  thesis. 
The  last  two  questions  in  the  introduction  were  eliminated  from  this  study  for  reasons 
discussed  in  the  methodology  chapter  and  are  not  discussed  here.  The  first  question 
contained  five  parts  that  asked  about  specific  system  changes  that  were  needed  for  the  trainer. 

1.  What  Changes  Need  to  Be  Made  in  the  Instructor's  Guide? 

The  instructor's  guide  provides  a  clear  and  concise  description  of  the  TPCT  system, 
but  it  needs  to  include  some  specific  direction  for  the  instructor.  The  guide  needs  to  include 
a  section  that  details  a  standard  pre-training  brief  to  ensure  continuity  among  instructors  and 
among  trainees.  Included  in  this  brief,  the  instructor  should  provide  the  trainee  with 
instruction  on  using  the  Integrated  Bridge  System  (DBS)  in  conjunction  with  the  trainer  clock 
to  solve  time  management  problems.  The  brief  should  also  contain  clarification  on  responding 
to  a  trainee's  request  for  information  during  the  scenario. 
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2.  What  Changes  Need  to  Be  Made  in  the  Student's  Guide? 

The  Student's  Guide  was  found  to  be  adequate  if  the  trainee  is  provided  with 
sufficient  time  to  study  it.  It  is  recommended  that  the  student  be  provided  with  instruction 
in  the  guide  on  using  the  trainer  clock  with  the  IBS  to  solve  time  management  problems. 

3.  What  Software  Problems  Need  to  Be  Corrected? 

The  software  operates  as  designed  and  the  one  noted  programming  problem  was 
corrected.  However,  various  recommendations  that  involve  software  changes  are  listed  as 
follows: 

•  Add  an  alternate  starting  point.  This  change  would  allow  the  trainee  to  begin 
a  second  or  third  session  closer  to  where  there  was  a  missed  decision  point, 
reducing  the  training  time,  making  it  possible  to  complete  the  scenario,  and 
aiding  the  perception  of  success. 

•  Add  a  "Permission  Denied"  button.  Although  this  button  would  not  affect  the 
scenario,  it  would  give  the  trainees  the  desired  physical  action  to  deny  a 
request  from  the  system  instead  of  simply  doing  nothing. 

•  Add  a  "demo  button"  to  allow  familiarity  with  the  user  interface. 

•  Add  steering  with  engines  capability.  This  added  feature  will  add  another 
element  of  realism,  allowing  the  trainee  to  take  an  action  that  is  available  on 
the  PC  class  ship. 

•  Adjust  fuel  to  indicate  a  minimum  of  10  percent  because  the  current  setting 
of  2  percent  is  not  realistic. 

•  Change  required  reduction  in  speed  during  fog.  The  speed  reduction  required 
by  the  trainee  should  be  in  accordance  with  standard  operating  procedures 
instead  of  an  arbitrary  amount. 

•  Add  temperature  changes  during  engine  casualties. 

•  Add  continuous  video  feed  and  visual  cues. 

•  Add  additional  scenarios. 
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4.  What  Hardware  Problems  Need  to  Be  Corrected? 

There  were  no  major  problems  noted  with  the  TPCT  hardware.  However  the 
recommendation  is  made  to  change  the  trainer  monitor  to  a  monitor  with  an  etched  finish  to 
reduce  glare.  Additionally,  the  hardware  needs  to  be  located  in  a  dedicated  training  area  that 
is  free  from  distractions. 

5.  What  Additional  Instruction  Needs  to  Be  Conducted  Prior  to  Using  the 
Trainer? 

The  TPCT  trainer  relies  heavily  on  the  use  of  the  IBS,  a  complicated  system  with 

which  the  trainees  do  not  have  familiarity.  It  is  recommended  that  a  prerequisite  to  TPCT 

training  course  of  instruction  be  developed  that  teaches  the  IBS.  The  IBS  component  of  the 

TPCT  can  be  used  to  conduct  this  training.  This  training  would  also  provide  the  trainee  with 

valuable  experience  on  a  system  that  will  be  on  that  trainees  ship.  Other  additional  instruction 

items  needed  were  noted  in  recommended  improvements  to  the  Instructor's  and  Student's 

guides. 

6.  What  Initial  Indications  of  System  Effectiveness  are  Produced  During 
Beta  Testing? 

No  absolute  measures  of  effectiveness  were  produced  by  this  research.  The  research, 

through  the  use  of  survey  instruments  and  trainee  perceptions,  generated  the  following 

indications  that  the  TPCT  operated  as  expected  or  was  perceived  as  effective. 

•  The  TPCT  induces  high  mental  demand. 

•  The  system  appears  to  generate  the  intended  stress  levels  in  trainees. 

•  The  human-system  interface  is  adequate  and  appropriate. 

•  The  system  is  relatively  easy  to  use. 
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•  Trainees  liked  the  system. 

•  Trainees  felt  the  system  was  realistic. 

•  Trainees  perceived  the  training  as  effective. 

•  Trainees  perceived  the  system  as  useful. 

B.  ADDITIONAL  FINDINGS  AND  RECOMMENDATIONS 

This  research  resulted  in  additional  findings  and  recommendations  that  were  not 
specific  to  the  original  research  questions.  The  following  list  summarizes  these  findings  and 
recommendations: 

•  Perceived  performance  and  success  in  the  training  do  not  appear  to  affect 
perceptions  of  training  utility.  Although  the  trainees  did  not  consider 
themselves  successful  in  the  scenario,  they  still  rated  the  TPCT  as  a  useful 
training  device. 

•  Most  prior  experience  with  computer-based  training  was  positive. 

•  It  is  recommended  that  the  system  be  adapted  to  train  other  watchstanders 
because  other  watchstanders  can  encounter  the  same  decision  making 
problems  that  the  TPCT  is  designed  to  enhance. 

•  It  is  recommend  that  the  TPCT  system  be  used  for  team  training,  especially 
bridge  watchteams  that  are  involved  in  assisting  the  CO  in  decision  making. 

•  It  is  recommend  that  the  technology  be  used  to  develop  trainers  for 
Engineering  Casualty  Control  (ECC),  ship  handling,  and  other  ship  types. 

•  It  is  recommended  that  the  use  of  voice  recognition  be  investigated  as  an 
alternative  user  interface. 

C.  RECOMMENDATION  STATUS 

During  the  evaluation  of  the  TPCT,  recommendations  were  submitted  to  Commander 
Naval  Special  Warfare  Command  (COMNAVSPECWAR.COM)  for  review.  The  following 
recommendations  were  approved  for  implementation: 
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D. 


Add  an  alternate  starting  point. 
Add  a  "Permission  Denied"  button. 

Add  a  "demo  button"  to  allow  familiarity  with  the  user  interface. 
Add  steering  with  engines  capability. 
Adjust  fuel  to  indicate  a  minimum  of  10  percent. 
Change  required  reduction  in  speed  during  fog. 
Add  temperature  changes  during  engine  casualties. 
RECOMMENDATIONS  FOR  FURTHER  RESEARCH 


Two  aspects  of  further  research  must  be  considered.  First,  any  changes  to  the  system, 
especially  software  additions,  need  to  undergo  continued  formative  evaluation  to  ensure  the 
system  works  properly.  Formative  evaluation  similar  to  that  conducted  in  this  thesis  is  also 
recommended  for  new  scenarios  or  applications  of  this  technology  to  other  areas  or  ship 
types.  It  is  also  recommended  that  further  research  be  conducted  on  the  TPCT  after 
implementation  to  determine  training  effectiveness. 

This  research  produced  initial  indicators  of  effectiveness.  Summative  evaluation  of 
the  TPCT  system  needs  to  be  conducted  to  check  actual  effectiveness.  Outcome-based 
measures  of  effectiveness  need  to  be  developed  and  tested  on  trainees  that  complete  a  course 
of  instruction  on  the  TPCT.  Additionally,  trainees  can  be  surveyed  at  a  later  time  to  gain 
information  on  perceived  utility  of  the  system. 


65 


66 


APPENDIX  A.  TRAINEE  DEMOGRAPHIC  SURVEY 


Mark  all  that  apply  (if  you  had  two  jobs  as  a  division  officer,  mark  them  both). 

Last  Name: 

First  Name: 

SSN: 

Age: 


Rank: 
YG: 


Education  Background 

BS  engineering  □ 

BS  non-engineering  □ 

BA  degree  □ 

MS  engineering  □ 

MS  non-engineering  □ 

MA  degree  □ 

PHD  □ 

Other:  □ 


(specify) 

Commissioning  Source 

USNA 

□ 

NROTC 

□ 

OCS 

□ 

ECP 

□ 

Seaman  to  Admiral 

□ 

Other: 

□ 

(specify) 


Ship  Types  Served  in 

CRUDES 

□ 

AMPHIB 

□ 

CV 

□ 

Auxiliary 

□ 

PC 

a 

MCM/MHC 

□ 

Other: 

□ 

(specify 

> 

Assignment  history 

Command  at  sea 

□ 

XO 

□ 

Department  Head 

OPS 

□ 

Deck 

□ 

Air 

□ 

Eng 

□ 

CSO 

G 

Weps 

□ 

Other: 

□ 

(specify) 

Division  Officer 

Eng 

□ 

Deck 

□ 

Air 

□ 

Ops 

□ 

Weps 

□ 

CS 

a 

Other: 

□ 

(specify) 
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APPENDIX  B.  PRE-TRAINING  INFORMATION  SURVEY 

1 .  Have  you  ever  used  an  interactive  training  simulator? 

Yes  □  No  □ 


2.  Only  answer  the  following  if  #1  was  "Yes" 

A.  Rate  your  experience: 

Negative 

1 2 3 — 


B.  Was  the  training: 

Ineffective 

1 2- 


Positive 
5 


Effective 
5 


C.  From  your  experience,  could  you  see  future  uses  for  that  type  of  training  in  the 


Navy? 


Yes  □ 


No  □ 
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APPENDIX  C.  POST  TRAINING  ORAL  QUESTIONS 

Ask  the  following  questions  immediately  after  completion  of  the  last  scenario  run  for  the 

student: 

Record  answers  in  the  space  below. 

1 .  Did  you  feel  pressured  to  complete  a  specific  task  during  the  mission? 

2.  Where  were  you  the  most  pressured? 


3.  Was  there  anything  you  needed  to  know  prior  to  starting  that  was  not  in  the  Student 
Guide  or  explained  by  the  instructor?  (What?) 


4.  Do  you  feel  as  if  you  have  learned  anything  new?(What?) 

5.  What  other  benefits  did  you  experience  from  using  the  TPCT? 

Administer  the  Post-test  surveys  (Sources-of- Workload  Evaluation,  User  Interface 
Dimensions,  and  the  TPCT  Post-Training  Information  survey)  ,  then  ask  questions  6  thru 
9. 

6.  Did  any  aspect  of  the  trainer  cause  you  to  feel  stressed? 

7.  What  aspect  caused  you  to  feel  stressed? 

8.  Do  you  consider  this  stress  a  positive  or  a  negative  aspect  of  the  trainer? 

9.  Is  there  anything  you  would  like  to  add,  recommendations  or  comments? 
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APPENDIX  D.  NASA  TASK  LOAD  INDEX 


The  evaluation  you  are  about  to  perform  is  a  technique  that  has  been  developed  by  NASA 
to  assess  the  relative  importance  of  six  factors  in  determining  how  much  workload  you 
experienced.  The  procedure  is  simple:  Read  the  following  task  descriptions  and  then  mark  the  scale 
at  the  point  that  reflects  the  task  load  that  you  experienced  after  completing  the  trainer.  If  you  have 
any  questions,  please  ask  them  now.  Thank  you  for  you  participation. 

1.  MENTAL  DEMAND 

A.  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking  searching,  etc.)? 

Low  High 

0123         4         5         6789        10 

B.  Was  the  task  easy  or  demanding,  simple  or  complex,  forgiving  or  exacting? 

Low  High 

0123         4         5         6789        10 


2.  PHYSICAL  DEMAND 

A.  How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)? 

Low  High 

0123         4         5         6789        10 

B.  Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful  or  laborious? 

Low  High 

0123         4         5         6789        10 

3.  TEMPORAL  DEMAND 

A.  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  tasks  or  task  elements 
occurred? 

Low  High 

0123         4         5         6789        10 

B.  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

Low  High 

0123         4         5         6789        10 
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4.  PERFORMANCE  DEMAND 

A.  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)? 

Low  High 

0123         4         5         6789        10 

B.  How  satisfied  were  you  with  your  performance  in  accomplishing  these  goals? 

Low  High 

0123         4         5         6789        10 

5.  EFFORT  DEMAND 

A.  How  hard  did  you  have  to  work  mentally  to  accomplish  your  level  of  performance? 

Low  High 

0123         4         5         6789        10 

B.  How  hard  did  you  have  to  work  physically  to  accomplish  your  level  of  performance? 

Low  High 

0123         4         5         6789        10 

6.  FRUSTRATION 

A.  How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure,  gratified,  content, 
relaxed  and  complacent  did  you  feel  during  the  task? 

Low  High 

0123         4         5         6789        10 
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APPENDIX  E.  USER  INTERFACE  DIMENSIONS 


Directions:  A  number  of  statements  that  describe  the  Interactive-multimedia  Courseware  (ICW) 
are  given  below.  Read  each  statement  and  then  circle  the  number  that  reflects  your  opinion.  There 
are  no  right  or  wrong  answers. 


Dimension  I  -  Ease  of  Use 

(Perceived  facility  with  which  user  interacts  with  the  ICW) 

Difficult  Easy 

0123         4         5         6789        10 


Dimension  2  -  Navigation 

(Pperceived  ability  to  move  through  the  contents  of  the  ICW) 

Difficult  Easy 

0123         4         5         6789        10 


Dimension  3  -  Cognitive  Load 

(Perceived  degree  that  the  user  interface  seems  manageable) 

Unmanageable  Manageable 

0123         4         5         6789        10 


Dimension  4  -  Mapping 

(Program's  ability  to  track  and  graphically  represent  user's  path  through  the  program) 

None  Powerful 

0123         4         5         6789        10 


Dimension  5  -  Knowledge  Space  Compatibility 

(Concepts  and  relationships  were  compatible  with  the  user's  knowledge  about  the  topic) 

Incompatible  Compatible 

0123         4         5         6789        10 
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Dimension  6  -  Information  Presentation 

(Perceived  degree  that  the  information  contained  in  the  ICW  is  presented  in  an  understandable 

form) 

Obtuse  Clear 

0123         4         5         6789        10 

Dimension  7  -  Media  Integration 

(How  much  does  the  ICW  coordinate  the  different  media  to  produce  an  effective  whole) 

Uncoordinated  Coordinated 

0123         4         5         6789        10 

Dimension  8  -  Overall  Functionality 

(Perceived  utility  of  the  ICW  in  relation  to  the  program's  intended  use) 

Dysfunctional  Highly  Functional 

0123         4         5         6789        10 
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APPENDIX  F.  POST-TRAINING  INFORMATION  SURVEY 


1.  Rate  how  difficult  or  easy  the  TPCT  system  was  to  operate 

Difficult                                                                                     Easy 
1 2 3 4 5 

2.  Rate  how  difficult  or  easy  the  touch  screen  was  to  operate. 

Difficult                                                                                     Easy 
1 2 3 4 5 

3.  Rate  the  TPCT  Scenario  according  to  the  following  criteria  (note:  This  question  refers 
to  the  battle  problem  itself  and  not  the  TPCT  system). 

Easy                                                                                        Difficult 
1 2 3 4 5 

Confusing                                                                                  Understandable 
1 2 3 4 5 

Slow  Fast 
1 2 3 4 5 

Unrealistic                                                                                  Realistic 
1 2 3 4 5 

4.  Rate  the  environment  according  to  the  following  criteria. 

Too  Too 
Cold  Hot 
1 2 3 4 5 

Very                                                                                           Very 
Uncomfortable                                                                           Comfortable 
1 2 3 4 5 

Too                                                                                            Too 
Quiet                                                                                          Loud 
1 2 3 4 5 
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5.  Rate  the  usefulness  of  the  TPCT  as  a  trainer  for  prospective  PC  class  Commanding 
Officers. 

Completely                                                                                 Very 
Worthless                                                                                Useful 
1 2 3 4 5 

6.  Rate  the  usefulness  of  this  type  of  simulation  for  other  decision  making  training 

Completely                                                                                 Very 
Worthless                                                                                   Useful 
1 2 3 4 5 


7.  Mark  any  area  that  caused  you  difficulty  while  using  the  TPCT. 
Q  Inputting  information  with  the  touch  screen  monitor 

□  Understanding  audio  reports 

□  Speed  or  volume  of  the  information  presented  (did  you  easily  lose  track  of  the 
situation  due  to  the  speed  or  volume  of  the  information  flow?) 

□  Inputting  information  into  the  Integrated  Bridge  System  (IBS) 

□  Other  (please  specify): 


A.  In  the  space  provided  below,  explain  why  the  items  you  checked  caused  you 
difficulty. 


8.  Rate  the  effectiveness  of  the  Student  Guide  (Did  it  prepare  you  for  operation  of  the 
TPCT?) 

Ineffective                                                                                  Effective 
1 2 3 4 5 

9.  Rate  the  effectiveness  of  the  instructor  in  providing  needed  information  prior  to 
operation  of  the  TPCT  system. 

Ineffective                                                                                  Effective 
1 2 3 4 5 
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10.  Did  the  instructor  provide  too  much  or  not  enough  information  during  the  scenario? 

Not  Enough                                                                               Too  Much 
1 2 3 4 5 


1 1 .  How  successful  do  you  feel  you  were  during  the  training  scenario? 

Not  Successful                                                                      Very  Successful 
1 2 3 4 5 

12.  What  problems  did  you  encounter  while  using  the  TPCT? 


13.  What  aspects  of  the  TPCT  did  you  like  the  most? 


14.  What  aspects  of  the  TPCT  did  you  like  the  least? 


!5.  What  recommendations  would  you  make  for  future  use  of  the  TPCT? 


16.  What  recommendations  would  you  make  for  future  use  of  similar  trainers? 
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APPENDIX  G.  FACILITATOR'S  GUIDELINES 


Directions  for  conduct  of  training  for  data  gathering  purposes.  The  following  is  a 
guideline  that  will  assist  the  facilitator  in  completing  all  necessary  information  for  research 
purposes. 

1.  Have  student  read  Student  Guide,  provide  15-30  minutes  if  needed,  note  the  amount  of 
time  the  student  actually  reads  the  guide  and  record  any  questions  asked. 

2.  Brief  student  on  system  IAW  instructor's  guide  and  include  the  following: 

"During  this  training  period  you  will  be  assisting  in  the  beta  test  process  of  the 
TPCT.  Your  cooperation  in  completing  pre  and  post  training  surveys  will  be 
greatly  appreciated.  The  information  gathered  in  the  surveys  is  for  research 
purposes  only,  to  assist  in  the  installation  of  the  TPCT.  In  no  way  will  your  inputs 
be  utilized  for  other  than  research.  Your  candid  comments  are  welcomed  and 
encouraged.  Your  performance  is  being  examined  for  the  purpose  of  evaluating 
the  TPCT  system." 

Have  student  fill  out  demographic  survey. 

Record  any  questions  asked  by  the  student  during  the  brief. 

3 .  Start  mission. 

Record  questions,  statements,  and  significant  actions  (i.e.,  Student  looks  confused, 
student  yells  "NO"  to  OOD,  student  says  "What  do  I  do  now?",  even  if  no  answer 
is  given). 

Record  any  assistance  given  to  the  student. 

4.  At  end  of  mission:  Debrief  the  mission,  explaining  missed  training  points,  answer 
student  questions,  and  record  these  items. 

5.  Run  mission  again  until  success  or  time  expiration,  continue  to  evaluate  the  training 
using  numbers  3  and  4  above  until  completion. 

6.  Administer  post-training  interview  and  surveys  after  the  final  debrief. 

7.  Collect  all  surveys,  notes  and  mission  evaluation  sheets  into  one  package  for  each 
student. 

8.  For  the  facilitator:  note  any  problems  encountered  in  administering  these  data  sheets: 
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